A developer's guide to building an AI security governance framework

Artificial Intelligence (AI) has firmly established itself as a pillar of digital transformation, disrupting industries, increasing efficiency, and providing unmatched access to large data sets. AI also raises profound questions regarding security governance. How do I ensure I can leverage the best of what AI has to offer while mitigating its potential security risks? As AI continues to advance, there is a growing need for strong oversight and accountability. This article delves into the complex landscape of AI security governance, exploring various frameworks, strategies, and practices that organizations like GitLab are adopting to ensure the responsible development of AI technologies and features.

Greater scrutiny on AI

AI: Single term, numerous realities

AI isn't a monolithic entity - it encompasses a spectrum of technologies and applications. From machine learning algorithms that power recommendation systems to advanced natural language processing models like Anthropic’s Claude 3, each AI system brings its unique set of opportunities and challenges.

According to a 2023 MITRE report, three main areas of AI currently exist:

1. AI as a subsystem

"AI is embedded in many software systems. Discrete AI models routinely perform machine perception and optimization functions, from face recognition in photos uploaded to the cloud, to dynamically allocating and optimizing network resources in 5G wireless networks.

"There are a wide range of vulnerabilities and threats against these types of AI subsystems – from data poisoning attacks to adversarial input attacks – that can be used to manipulate subsystems."

2. AI as human augmentation

"Another application of AI is in augmenting human performance, allowing a person to operate with much larger scope and scale. This has wide-ranging implications for workforce planning as AI has the potential to increase productivity and shift the composition of labor markets, similar to the role of automation in the manufacturing industry.

"While sophisticated hackers and military information operations can already generate believable content today using techniques such as computer-generated imagery, LLMs will make that capability available to anyone, while increasing the scope and scale at which the professionals can operate."

3. AI with agency

"A segment of the tech community is increasingly concerned about scenarios where sophisticated AI could operate as an independent, goal-seeking agent. While science fiction historically embodied this AI in anthropomorphic robots, the AI we have today is principally confined to digital and virtual domains.

"One scenario is an AI model given a specific adversarial agenda. Stuxnet is perhaps an early example of sophisticated, AI-fueled, goal-seeking malware with an arsenal of zero-day attacks that ended up escaping onto the internet."

You can focus your efforts in terms of security governance based on which areas your company is looking to adopt and the expected business benefits.

Frameworks for AI security governance

For effective AI security governance, we must navigate the complex landscape of guidelines and principles developed by various organizations.

Governments, international organizations, and tech companies have all played their part in shaping AI security governance frameworks. You can review the frameworks below and choose those that are relevant and/or apply to your organization:

• NIST AI Risk Management Framework (AI RMF)
• Google’s Security Artificial Intelligence Framework
• OWASP Top 10 for LLMs
• The UK’s NCSC Principles for the Security of Machine Learning

While these frameworks provide valuable guidance, they also introduce complexity. Organizations must determine which apply to their AI usage and how they align to their practices. Moreover, the dynamic nature of AI requires continuous adaptation to stay secure.

Something to note is that if you read through these frameworks, you’ll notice that numerous controls overlap with standard security best practices. This isn’t a coincidence. A strong overall security program is a prerequisite for proper AI security governance.

How-to: AI security governance

The why and the what

AI security governance starts with understanding what AI technologies your organization is using or developing, why you are using them, and where these technologies fit into your operations. It's essential to define clear objectives and identify potential security risks associated with AI deployment. This introspection lays the foundation for effective AI security governance.

The why

Understanding the "why" behind each AI application is pivotal to build effective security governance. Each AI system deployed has to serve a specific purpose. Is AI being utilized to enhance customer experiences, automate manual tasks, or support the decision-making process?

By uncovering the motivations driving AI initiatives, organizations can align these projects with their broader business objectives. This alignment ensures that AI investments are strategically focused, delivering value in line with organizational goals. It also aids in prioritizing AI systems that have a more significant impact on the core mission of the company.

The what

In the realm of AI security governance, the foundational step is conducting a comprehensive inventory of all AI systems, algorithms, and data sources within your organization. This includes meticulously cataloging all AI technologies in use, ranging from machine learning models and natural language processing algorithms to computer vision systems. This would also involve identifying the data sources feeding these AI systems, and their origins (internal databases, customer interactions, or third-party data providers). Such an inventory provides three main benefits:

• to gain a holistic understanding of the AI ecosystem within the organization
• to establish a strong basis for monitoring, auditing, and managing these assets effectively
• to focus security efforts on the high-risk/critical areas

How to develop a security risk management program

A robust security risk management program is at the core of responsible AI security governance. The critical building blocks for this program are the what and the why we discussed earlier.

Specificities of AI make security risk management more complex. In the NIST AI RMF mentioned earlier, numerous challenges are highlighted, including:

• Difficult to measure AI-related security risks
  • Potential security risks could emerge from the AI model, the software on which you are training the model, or the data ingested by the model. Different stages of the AI lifecycle might also trigger specific security risks depending on which actors (producers, developers, or consumers) are leveraging the AI solution.
• Risk tolerance threshold might be complex to determine
  • As the potential security risks aren’t easily identifiable, determining the risk tolerance your organization can withstand regarding AI can be a very empirical exercise.
• Not considering AI in isolation
  • Security governance of AI systems should be part of your security risk management strategy. Different users might have different parts of the overall picture. Ensuring you have complete information and full visibility into the AI lifecycle is critical to making the best decisions.

Security risk management should be an ongoing process, adapting to the quickly evolving AI landscape. Reassessing the program, reviewing assumptions regarding the environment and involving additional business stakeholders are activities that should be happening on a regular basis.

AI security governance and the GitLab DevSecOps platform

Using AI to power DevSecOps

Let’s take GitLab Duo, our suite of AI capabilities to help power DevSecOps workflows, as an example. GitLab Duo Code Suggestions helps developers write code more efficiently by using generative AI to assist in software engineering tasks. It works either through code completion or through code generation using natural language code comment blocks.

To ensure it can be fully leveraged, security needs of potential users and customers have to be considered. As an example, data used to produce Code Suggestions is immediately discarded by the AI models.

All of GitLab’s AI providers are subject to contractual terms with GitLab that prohibit the use of customer content for the provider’s own purposes, except to perform their independent legal obligations. GitLab’s own privacy policy prevents us from using customer data to train models without customer consent.

Of course, to fully benefit from Code Suggestions, you should:

• understand and review all suggestions to see if they align with your development guidelines
• limit providing sensitive information or proprietary code in prompts ensure the suggestion follows the same secure coding guidelines your company has
• review the code using automated scanning for vulnerable dependencies, input validation and output sanitization, as well as license checks

Securing AI

Managing the output of AI systems is equally important as managing the input. Security scanning tools can help identify vulnerabilities and potential threats in AI-generated code.

Managing AI output requires a systematic approach to code review and validation. Organizations should integrate security scanning tools into their CI/CD pipelines, ensuring that AI-generated code is checked for security vulnerabilities before deployment. Automated security checks can help detect vulnerabilities early in the development process, reducing the risk of potential vulnerable code stemming from suggested code blocks being merged.

For any GitLab Duo generated code, changes are managed via merge requests which trigger your CI pipeline (including any security and code quality scanning you have configured). This ensures any governance rules you have set up for your merge requests like required approvals are enforced.

AI systems are systems. Existing security controls apply to AI systems the same way they would apply to the rest of your environment. Common security controls around application security still apply, including security reviews, security scanning, threat modeling, encryption, etc. The Google Secure AI Framework highlights these six elements:

• expand strong security foundations to the AI ecosystem
• extend detection and response to bring AI into an organization’s threat universe
• automate defenses to keep pace with existing and new threats
• harmonize platform-level controls to ensure consistent security across the organization
• adapt controls to adjust mitigations and create faster feedback loops for AI deployment
• contextualize AI system risks in surrounding business processes

If you have a strong security program, managing AI will be an extension of your current program and account for specific risks and vulnerabilities.

How GitLab Duo is secured

GitLab recognizes the significance of security in AI governance. Our very strong security program is focused on ensuring our customers can fully leverage GitLab Duo in a secure manner. This is how the security departments are collaborating to secure GitLab’s AI features GitLab:

• Security Assurance: Seeks to address our compliance requirements regarding security, that AI security risks are identified and properly managed, and that our customers understand how we secure our application, infrastructure, and services.

• Security Operations: Monitors our infrastructure and quickly responds to threats using a team of skilled engineers as well as automation capabilities, helping to ensure AI features aren’t abused or used in a malevolent manner.

• Product Security: Helps the product and engineering teams by providing security expertise for our AI features and helping to secure the underlying infrastructure on which our product is hosted.

• Corporate Security and IT Operations: Finds potential vulnerabilities in our product to proactively mitigate and support other departments by performing research on relevant security areas.

Our Security team works closely with GitLab's Legal and Corporate Affairs team to ensure our framework for AI security governance is comprehensive. The recent launch of the GitLab AI Transparency Center showcases our commitment to implementing a strong AI governance. We published our AI ethics principles as well as our AI continuity plan to demonstrate our AI resiliency.

Learn more

AI security governance is a complex area, especially as the field is in a nascent form. As AI continues to support our workflows and accelerate our processes, responsible AI security governance becomes a key pillar of any security program. By understanding the nuances of AI, enhancing your risk management program, and using AI features that are developed responsibly, you can ensure that AI-powered workflows follow the principles of security, privacy, and trust.

Learn more about GitLab Duo AI features.