Own your AI: Self-Hosted GitLab Duo models with AWS Bedrock

As organizations adopt AI capabilities to accelerate their software development lifecycle, they often face a critical challenge: how to leverage AI while maintaining control over their data, infrastructure, and security posture. This is where GitLab Duo Self-Hosted provides a compelling solution. In this article, we'll walk through the implementation of GitLab Duo Self-Hosted models. This comprehensive guide helps organizations needing to meet strict data sovereignty requirements while still leveraging AI-powered development. The focus is on using models hosted on AWS Bedrock rather than setting up an LLM serving solution like vLLM. However, the methodology can be applied to models running in your own data center if you have the necessary capabilities.

Why GitLab Duo Self-Hosted?

GitLab Duo Self-Hosted allows you to deploy GitLab's AI capabilities entirely within your own infrastructure, whether that's on-premises, in a private cloud, or within your secure environment.

Key benefits include:

• Complete Data Privacy and Control: Keep sensitive code and intellectual property within your security perimeter, ensuring no data leaves your environment.
• Model Flexibility: Choose from a variety of models tailored to your specific performance needs and use cases, including Anthropic Claude, Meta Llama, Mistral families, and OpenAI GPT families.
• Compliance Adherence: Meet regulatory requirements in highly regulated industries where data must remain within specific geographical boundaries.
• Customization: Configure which GitLab Duo features use specific models to optimize performance and cost.
• Deployment Flexibility: Deploy in fully air-gapped environments, on-premises, or in secure cloud environments.

Architecture overview

The GitLab Duo Self-Hosted solution consists of three core components:

1. Self-Managed GitLab instance: Your existing GitLab instance where users interact with GitLab Duo features.
2. AI Gateway: A service that routes requests between GitLab and your chosen LLM backend.
3. LLM backend: The actual AI model service, which, in this article, will be AWS Bedrock. Note: You can use another serving platform if you are running on-premises or using another cloud provider.

Prerequisites

Before we begin, you'll need:

• A GitLab Premium or Ultimate instance (Version 17.10 or later)

  • We strongly recommend using the latest version of GitLab as we continuously deliver new features.

• A GitLab Duo Enterprise add-on license

• AWS account with access to Bedrock models or your API key and credentials needed to query your LLM Serving model

Note: If you aren't a GitLab customer yet, you can sign up for a free trial of GitLab Ultimate, which includes GitLab Duo Enterprise.

Implementation steps

1. Install the AI Gateway

The AI Gateway is the component that routes requests between your GitLab instance and your LLM serving infrastructure — here that is AWS Bedrock. It can run in a Docker image. Follow the instructions from our installation documentation to get started.

For this example, using AWS Bedrock, you also must pass the AWS Key ID and Secret Access Key along with the AWS region.

AIGW_TAG=self-hosted-v18.1.2-ee`
docker run -d -p 5052:5052 \

  -e AIGW_GITLAB_URL=<your_gitlab_instance> \

  -e AIGW_GITLAB_API_URL=https://<your_gitlab_domain>/api/v4/ \

  -e AWS_ACCESS_KEY_ID=$AWS_KEY_ID

  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \

  -e AWS_REGION_NAME=$AWS_REGION_NAME \

registry.gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/model-gateway:$AIGW_TAG \

Here is the AIGW_TAG list.

In this example we use Docker, but it is also possible to use the Helm chart. Refer to the installation documentation for more information.

2. Configure GitLab to access the AI Gateway Now that the AI gateway is running, you need to configure your GitLab instance to use it.

• On the left sidebar, at the bottom, select Admin.

• Select GitLab Duo.

• In the GitLab Duo section, select Change configuration.

• Under Local AI Gateway URL, enter the URL for your AI gateway and port for the container (e.g., https://ai-gateway.example.com:5052).

• Select Save changes.

3. Access models from AWS Bedrock

Next, you will need to request access to the available models on AWS Bedrock.

• Navigate to your AWS account and Bedrock.

• Under Model access, select the models you want to use and follow the instructions to gain access.

You can find more information in the AWS Bedrock documentation.

4. Configure the self-hosted model Now, let's configure a specific AWS Bedrock model for use with GitLab Duo.

• On the left sidebar, at the bottom, select Admin.

• Select GitLab Duo Self-Hosted.

• Select Add self-hosted model.

• Fill in the fields:

  • Deployment name: A name to identify this model configuration (e.g., "Mixtral 8x7B")
  • Platform: Choose AWS Bedrock
  • Model family: Select a model, for example here "Mixtral"
  • Model identifier: bedrock/model-identifier from the supported list.

• Select Create self-hosted model.

5. Configure GitLab Duo features to use your self-hosted model

After configuring the model, assign it to specific GitLab Duo features.

• On the left sidebar, at the bottom, select Admin.

• Select GitLab Duo Self-Hosted.

• Select the AI-powered features tab.

• For each feature (e.g., Code Suggestions, GitLab Duo Chat) and sub-feature (e.g., Code Generation, Explain Code), select the model you just configured from the dropdown menu.

For example, you might assign Mixtral 8x7B to Code Generation tasks and Claude 3 Sonnet to the GitLab Duo Chat feature. Check out the requirements documentation to select the right model for the use case from the models compatibility list per Duo feature.

Verifying your setup

To ensure that your GitLab Duo Self-Hosted implementation with AWS Bedrock is working correctly, perform these verification steps: 1. Run the health check After running the health check of your model to be sure that it’s up and running, Return to the GitLab Duo section from the Admin page and click on Run health check. This will verify if:

• The AI gateway URL is properly configured.
• Your instance can connect to the AI gateway.
• The Duo Licence is activated.
• A model is assigned to Code Suggestions — as this is the model used to test the connection.

If the health check reports issues, refer to the troubleshooting guide for common errors.

2. Test GitLab Duo features Try out a few GitLab Duo features to ensure they're working:

• In the UI, open GitLab Duo Chat and ask it a question.
• Open the web IDE
  • Create a new code file and see if Code Suggestions appears.
  • Select a code snippet and use the /explain command to receive an explanation from Duo Chat.

3. Check AI Gateway logs Review the AI gateway logs to see the requests coming to the gateway from the selected model: In your terminal, run:

docker logs <ai-gateway-container-id>

Optional: In AWS, you can activate CloudWatch and S3 as log destinations. Doing so would enable you to see all your requests, prompts, and answers in CloudWatch. Warning: Keep in mind that activating these logs in AWS logs user data, which may not comply with privacy rules. And here you have full access to using GitLab Duo's AI features across the platform while retaining complete control over the data flow operating within the secure AWS cloud.

Next steps

Selecting the right model for each use case

The GitLab team actively tests each model's performance for each feature and provides tier ranking of model's performance and suitability depending on the functionality:

• Fully compatible: The model can likely handle the feature without any loss of quality.
• Largely compatible: The model supports the feature, but there might be compromises or limitations.
• Not compatible: The model is unsuitable for the feature, likely resulting in significant quality loss or performance issues. As of this writing, most GitLab Duo features can be configured with Self Hosted. The complete availability overview is available in the documentation.

Going beyond AWS Bedrock

While this guide focuses on AWS Bedrock integration, GitLab Duo Self-Hosted supports multiple deployment options:

1. On-premises with vLLM: Run models locally with vLLM for fully air-gapped environments.
2. Azure OpenAI Service: Similar to AWS Bedrock, you can use Azure OpenAI for models like GPT-4.

Summary

GitLab Duo Self-Hosted provides a powerful solution for organizations that need AI-powered development tools while maintaining strict control over their data and infrastructure. By following this implementation guide, you can deploy a robust solution that meets security and compliance requirements without compromising on the advanced capabilities that AI brings to your software development lifecycle. For organizations with stringent security and compliance needs, GitLab Duo Self-Hosted strikes the perfect balance between innovation and control, allowing you to harness the power of AI while keeping your code and intellectual property secure within your boundaries. Would you like to learn more about implementing GitLab Duo Self-Hosted in your environment? Please reach out to a GitLab representative or visit our documentation for more detailed information.