In February 2020, Harvard University and the Linux Foundation’s Core Infrastructure Initiative released a joint report, Vulnerabilities in the Core, looking at security challenges in the open source software world. Open source software has taken over the world, but with its astronomical popularity comes the potential for huge risk. We thought this was an excellent opportunity to ask senior security engineer Mark Loveless for his thoughts on open source security, how GitLab approaches it, and some ways you can move the bar forward in your company.
I was quite pleased that an institution like Harvard would be taking a hard look at open source software. Security is often difficult for people to understand how much it impacts real lives. Nerd types (the InfoSec community) try to communicate to the "normals" to explain to them why they need to care about security. Granted, this barely tracks on the radar of many people in our own field, as some of the Infosec community think they already know it.
Inside our security plan
We have dedicated people at GitLab looking at our own code, trying to find security flaws. We have a group that deals with bug submissions coming in from Hacker One. But we also have people at GitLab looking for security flaws in various open source packages that are a part of the "supply chain". These outside packages may be used in the GitLab product. When one of these packages has a flaw in it, there can be impact to both open source projects like ours, as well as closed source projects. It may be surprising to learn that a lot of closed source projects use open source libraries.
Using an open source library to complete a coding project is not uncommon and in fact highly encouraged. For example, reading through the Internet standards for protocol implementation of HTTPS could not only be daunting, but coding it without the experience of writing security-related code is ill-advised unless you are an expert in the field. Including a security-related open source library in your project can solve that issue. But speaking of something as complex as HTTPS, for example, brings up another problem - implementation.
The implementation issue
A flaw is found in an open source package. Hundreds of applications are using the open source package with the vulnerability, yet only half of these applications are exploitable via the flaw. Why? A lot of times it comes down to implementation.
When given a choice between one setting versus another, there are ramifications to consider when deciding which might make the most secure implementation. These choices have consequences. One choice could impact performance - especially at scale. Another choice could leave things more vulnerable. There are compliance issues to consider. All of these items need to be weighed carefully.
It’s a core thing at GitLab to be sure that we implement things in ways that are more secure. I’ve worked at numerous software vendors over the years and GitLab is one of the largest. Most organizations of this size do not release code nearly as often as we do, and most do not take security as seriously. When I started working here, I was pleasantly surprised at how much security was part of the process.
Trust the transparency
We’re not only a transparent company, we’re an open core company. This means that the core part of our product is open source and free to download and use, while we have a number of paid options for increased features and services. This applies to everything in our company and is an added benefit when it comes to security. We’re very open about disclosing security problems. For example, when a bug affects our open source code any future code commit to fixing the problem also shows the vulnerability. At some of my other employers there were often discussions about the "exposure" of the company when revealing a vulnerability, and a pull between departments about how much to disclose. Sometimes those disclosure discussions turned political. At GitLab that problem is eliminated as we’ve made the commitment to completely disclose the issue. We’ve even extended this to the parts of our offerings that are not a part of that core product. It is in the best interest of our users to have complete information about the security of GitLab.
The role DevSecOps plays
When it comes to writing code, bugs - security-related or not - are the nature of the beast. The trick is to expect it and plan for it. Here at GitLab we’re doing the DevSecOps thing: extraordinarily rapid development while retaining a focus on security. There is a tendency here for our engineers to take the time and do things right. Bugs can impact performance and availability and security bugs are no different. This is why GitLab works so hard on delivering code that is as clean and secure as possible. When we manage to do that it’s a win/win. When bugs do occur, we have developed processes to deal with them which includes updates to our development processes if necessary to help improve the entire process.
Game on
If you’re trying to up your open source security game you already know it’s a constant work in progress. We encourage open source as much as possible! Open source or not, here are a few things that we’ve done at GitLab that apply to security:
- You have to have security people involved in the company. Any security team needs to have people who have the capacity to analyze and handle flaws in code, whether the code is internal or external.
- The security people need to be on staff. They need to have the freedom to do their jobs.
- Upper management has to buy into security. If security is top of mind for upper management it will be easier to get things done at the department level.
- Lose the culture of shame and punishment for security problems. It exists in lots of places and it’s why people start trying to hide bugs. There should not be a carrot and a stick involved in security.
- Make security something that’s muscle memory and you’ll be golden.
Cover image by Wolfgang Hasselmann on Unsplash
