Why we implemented our own SSHD solution

Aug 17, 2022 · 5 min read
Igor Drozdov GitLab profile

The story of why we moved to our own SSHD is an interesting one. GitLab provides a number of features that execute via an SSH connection. The most popular one is Git-over-SSH, which enables communicating with a Git server via SSH. Historically, we implemented the features through a combination of OpenSSH Server and a separate component, a binary called GitLab Shell. GitLab Shell processes every connection established by OpenSSH Server to communicate data back and forth between the SSH client and the Git server. The solution was battle-tested, and relied on a trusted component such as OpenSSH. Here's why we decided to implement our own SSHD.

Community contribution

Everyone can contribute at GitLab! A community contribution from @lorenz, gitlab-sshd, was suggested as a lightweight alternative to our existing setup. A self-contained binary with minimal external dependencies would be beneficial for containerized deployments. A GitLab-supported replacement also opened up new opportunities:

The previous architecture

Old architecture

Our current architecture

New architecture

Risks and challenges

However, changing a critical component that is broadly used, and is responsible for security, carries tremendous risks. We experienced both challenges and risks:

Problems we encountered

A component with a scope this broad could have a wide range of problems. We encountered, and resolved, the following problems:

Lessons learned

Unfortunately, issues became visible on production environment, thanks both to the load and the variety of possible OpenSSH configurations. Even though we caught some bugs on our staging environments, predicting all types of problems was almost impossible. However, these actions helped us resolve the issues:

“@gitlab implemented our own SSHD solution. Here's why we did it and what we learned” – Igor Drozdov

Click to tweet

Edit this page View source