Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Design :: Kubernetes Traffic Transition

Idea/Problem Statement

Currently, we utilize haproxy as our gateway into the GitLab.com product. Here we define multiple ACL's [[1]] [[2]] that handle a lot of control. Whether it be blocking abusers, slowing down potential DDoS, or adjusting how traffic is flowing through haproxy, it's responsible for quite a bit [[3]] [[4]]. We need to determine how to port over this into a kubernetes environment. GitLab.com may be deployed behind the provided nginx ingress from our own helm charts with none of these security controls. This will not work for GitLab.com. There are too many dependencies with how we route traffic to specific endpoints, the division of fleets to handle specific types of requests, and automation we have to assist us with security related concerns.

Design

As proven in a PoC done in [issue/6673] we will utilize our existing haproxy infrastructure and simply proxy the traffic to Kubernetes Ingress endpoints.

Implementation Considerations

Testing

This has already been tested in a manual format on our pre environment for the Container Registry service.

Operational Considerations

Configuration

We'll need a few items done to our chef cookbooks to allow for such a configuration:

  1. Modifications to add a custom configuration to allow healthchecks and traffic to pass to the desired service inside of kubernetes
  2. Additions of weights such that we can configure, when desired, how much traffic we are sending to Kubernetes vs VMs.

Issues for the above can be found linked to [issue/6673]

Additional Considerations

Please note that this is only a transitional project. Once we've completed the migration of a service to Kubernetes, we may change out how we send traffic to Kubernetes in the future. We've briefly spoken about utilizing Envoy as a better solution. It would be wise to remove ourselves of Virtual Machines as much as we can, as stateless load balancers are great candidates for taking advantage of the Kubernetes platform. This has been considered a new project and will warrant it's own Epic and set of Design Documents in the future.

[[1]]: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gprd-base-lb-fe-common.json [[2]]: https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/blob/master/templates/default/haproxy-frontend.cfg.erb [[3]]: https://gitlab.com/gitlab-com/security-tools/recaptcha [[4]]: https://gitlab.com/gitlab-com/security-tools/front-end-security [issue/6673]: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6673