Security provider KnowBe4 keeps code in-house and speeds up deployment

KnowBe4, Inc. is the provider of the world’s largest security awareness training and simulated phishing platform. KnowBe4 helps manage the ongoing problems of social engineering with on-demand, interactive training for tens of thousands of organizations worldwide. It was ranked highest in ‘Ability to Execute’ and ‘Completeness of Vision’ on the 2019 Gartner Magic Quadrant for Security Awareness CBT.

Members of KnowBe4’s engineering team were using three separate tools in their deployment toolchain. One for code management, another for code testing, and a third for code deployment. With three solutions in use, all of their work was spread out in different places. While each tool had its own value set, the lack of integration among them caused additional work and stress for users. “Deploys would go off and trigger jobs in multiple different tools. Tests would trigger in one location, the deployments in another. Context switching was constant, and due to everything running concurrently, you never achieved the correct continuous pipeline,” says Alex Callihan, VP of Platform Engineering at KnowBe4. “Tests had the potential to fail after a deploy already succeeded. This was a problem. With GitLab, we were able to consolidate this process into a single tool and ensure the pipeline was executed in order.”

The team also was looking to reduce the costs associated with operating three platforms per toolchain. With their code testing tool, each concurrent test capacity incurred an additional cost, so teams couldn’t scale as much as they wanted without considering the financial burden that comes with adding capacity. “With our old code testing tool, we had to provision to our maximum. So if we ever needed to run 50 concurrent tests, we were forced to pay for 50 all day, every day. Our cost was approximately $50 per concurrent test per month for 50 concurrent tests, even though outside of our core business hours the tests were rarely needed to that magnitude,” says Matthew Duren, Principal Site Reliability Engineer.

KnowBe4 was looking to consolidate to one tool that could provide end-to-end visibility. If the team no longer had to to spend time context switching between various tools, deployment speed could soar automatically. Other priorities for a new toolset included being able to:

• Be self-managed
• Reside in AWS
• Integrate with Jira, Docker, and other tools as needed
• Scale their Git solution in tandem with their employee and service growth

Duren and Callihan had previously used GitLab’s free version. Their experience and understanding of the platform was a driving factor in bringing GitLab to stakeholders.

The pair was eager for the rest of the company to understand the scope of the platform’s capabilities. In order to do that, they chose to do a proof of concept with a single product from their project list. The team was obligated to deliver the product in one month. “We happened to pick one product that we had to ship in a month. We shipped it on GitLab while we were still in the trial period,” Callihan said.

For stakeholders, an important aspect was ensuring security by keeping code in-house, which GitLab enabled them to do. “As our company has grown, being compliant across multiple standards has been a key goal of our development and infosec teams. GitLab’s tooling for security and ability to host within our own infrastructure was a big selling point to get approval for the POC.” Callihan says.

GitLab is at the heart of KnowBe4’s software development lifecycle. A developer will first open a feature branch off of master in GitLab. From there, they can deploy an on-demand development environment leveraging GitLab pipelines. When the development environment is determined verified working by QA, the developer opens a merge request to master. All commits then run test pipelines until that merge request is approved and ultimately merged. After merge, a pipeline is started to build and release the Docker image to AWS. After the release, a deployment stage kicks off leveraging Terraform to roll-in the latest image into production. All of this is orchestrated by GitLab runners deployed in AWS with full logging and visibility. Production now deploys five or more times per day for any given application. Development environments deploy 20 or more times per day for any given application. Hundreds to thousands of test jobs run every day across all applications.

The teams have standardized the development lifecycle for more than 60 microservices. “Due to that standardization, the simplicity of starting new projects or troubleshooting existing ones is incredibly easy. We all know how projects will build, release, and ship regardless of codebase or design,” Callihan says. “With a healthy mix of Docker, Terraform, and GitLab in GitLab pipelines, we’ve got a system in place that is super efficient.” They’ve also lowered time to production by allowing multiple concurrent test pipelines and with deploying and auto-scaling their own runners.

GitLab’s continuous integration helps prevent bugs before code hits production. Security modules are built into the test pipelines and count as failed tests, which then cancel pipelines, protecting against deploying any vulnerable code.

Since KnowBe4 already had a modern architecture in place, GitLab was able to help the company ship faster and solve unique use cases. All the teams now collaborate within the same tooling whether for code reviews, pipelining, or code ownership. The adoption was straightforward, given the simplicity of the tool and GitLab YMLs were fully embraced by all teams.

“Moving test, build, and deployment tooling into the application repositories themselves has given developers the opportunity to build and manage their own deployment pipelines, reducing the responsibility of the SRE team to simply review and approve the developers’ changes,” Duren says. “By making this relatively straightforward adjustment, we are able to deploy changes much more frequently than we otherwise would have been able to do. We removed a huge bottleneck, which sped up our development lifecycle and improved continuity between our team and the rest of R and D.”