#sandbox-cloud-questions Slack channel to ask questions and get help.The Sandbox Cloud is an automated provisioning platform for creating an AWS account or GCP project that you can use for demo/sandbox/testing purposes and is paid for by GitLab with consolidated invoice billing (no credit card required).
This platform is powered by HackyStack, an open source project created by Jeff Martin and maintained by Jeff Martin and Dillon Wheeler to automate the access request process using Okta integration, auto-assigning roles and permissions based on your department, and using the cloud provider API for provisioning your AWS account and/or GCP project.
You can learn more in the HackyStack High-Level Intro presentation.
The Sandbox Cloud is managed by the IT Engineering team. Please tag Jeff Martin in Slack with any questions.
We are currently at
betastability. Please post in#sandbox-cloud-questionsif you see unexpected behavior.
Any team member can use the self service instructions below to provision an AWS account or GCP project for your individual use (sandbox, testing, etc.). You cannot invite other team members to individual accounts for security reasons.
#sandbox-cloud-questions to have it added.Provisioning to Active.You can sign-in with Okta, however please don’t create a Cloud Account unless you intend to provision AWS resources. You can see the screenshots of everything that a user sees.
Is your current AWS account experience problems? Please ask for help in #sandbox-cloud-questions. If your problems are validated and approved for getting a new AWS account, please use the New AWS Individual Account Rebuild Request issue template.
For cost saving and exposure reduction measure, any GCP compute engine instances in an individual account (ex. dmurphy-a1b2c3d4) are powered down every Friday at 23:59 UTC starting 2023-02-03. You will receive a Slack bot notification 24 hours in advance for any instances that are affected with instructions for adding a label to the instance if you need to prevent power off.
If an instance is powered off, you can simply power the instance back on when you're ready to work again.
We are not charged for the hours that compute instances that are not powered on (e2-standard-4/4vCPU/16GB costs $0.13/hr or $96.48/mo). Storage for a powered down instance is cheap and a 20GB persistent disks cost $0.80/month.
This has immediate 28%+ cost savings by not having ephemeral infrastructure running on Saturdays and Sundays. Additional cost savings since we are not charged until you power the instance back on (could be days, weeks, or months later). This also covers abandoned instances that were only used for a few hours for a demo.
We will be working on AWS accounts in a future iteration.
Any team member can request a new AWS account or GCP project for a specific project or working group, or for general usage by their department group for shared non-production resources.
No RED data is allowed in these accounts/projects. Any RED data must be hosted in production AWS accounts or GCP projects managed by the appropriate Infrastructure Realm administrators (ex. eng-infra-saas, it-infra, etc.).
Self-service creation and IAM management is not available yet for end users in HackyStack (will be available through API integration with GitLab Access Manager in the future and tracked in hackystack#38). In the meantime, we use access request style issue templates as our boring solution for security compliance reasons and the HackyStack administrators provision accounts and users using the Admin CLI.
For any staging or production(-esque) infrastructure services that are customer facing, contain Red or Orange data, related to the GitLab product or GitLab.com SaaS, or Engineering sponsored services, please contact the Reliability Engineering team for guidance on next steps in the #infrastructure_lounge Slack channel.
Most environments are typically created in the config-mgmt project using the Create a new environment instructions.
You can learn more about GitLab.com SaaS on the Production Architecture handbook page.
Any projects with yellow or green data usually are better suited for self management using Group Projects using Infrastructure Standards guidelines.
For any infrastructure services related to business operations and our tech stack, please contact the IT team in #it_help for guidance on next steps. Most of our tech stack are SaaS-based and hosted by the respective vendor.
New SaaS applications should go through the Procurement Process and are managed by the respective department's system owners.
Self-hosted application infrastructure is determined on a case-by-case basis and is architected in collaboration with IT Infrastructure, Security Architecture, Infrastructure Security, Application Security, and 3rd Party Risk. Please tag @jeffersonmartin in an issue for preliminary guidance on new services. If you do not have an issue yet, please create one in the IT Infrastructure issue tracker.
Cloud Account details page, click the View IAM Credentials button in the top right corner to open up a popup modal window.AWS Console URL, Username, and Password that you can use to sign in to your AWS account. The 12 digit number at the beginning of the URL is your AWS Account ID/Number.Open AWS Web Console button on the Cloud Account details page.AdministratorAccess to be able to perform any action inside of your AWS account. We do not provide team members access to the root user account since we only use this for break glass security incidents or related administrative activity by the Infrastructure Realm Owners.Cloud Account details page, click the View Credentials button in the top right corner to open up a popup modal window.GCP Console URL and username. Since GCP uses Google authentication, you simply need to be signed in with your GitLab email address on Google. HackyStack has added the Owner GCP IAM role to your email address when the project was created. Your project ID is in the format of {emailHandle}-{cloudAccountShortId}. You can choose this project from the dropdown list when accessing a different project in the GCP console.See the Domain Names and DNS Records IT guide handbook page for more details.
In the HackyStack v1.11 (November 2021) release, we introduced Terraform environment generation for GCP projects and a lot of underlying automation for GitOps with alpha stability. AWS will be supported in a future iteration.
WAUBVZA3bneq4oWx)
gcp-sandbox-environment-template-v2-######## template.{firstInitial}{lastName}-{hash} and not your normal GitLab username.Plan job completes, trigger the Deploy job. (Notice how you haven’t had to do any configuration).terraform apply outputs as your new environment is spun up with a sample Ubuntu virtual machine for testing with. You can add additional Terraform resources as you see fit (see below).gcloud command or Cloud Shell.Destroy to clean up your resources.terraform/main.tf file in the Git repository to add more Terraform resources or modules.Deploy CI pipeline job to deploy your resources.Currently, deleting an AWS account or GCP project must be performed manually by IT Ops. This is done during offboarding for operational consistency. However, outside of offboarding, you must make a best effort to delete all resources within the account yourself. There may be a nominal monthly cost of a few dollars a month to keep the account in existence. At this time, this cost is deemed acceptable.
The oversimplified user story is "I need to spin up VM(s) or cluster(s) in GCP or AWS to try something (anything, may not be GitLab product specific), what's the company infrastructure standards for doing that?"
The goal is to create a frictionless approach for technical team members that includes the tagging needed for cost allocation, best practice security configurations, and streamline the provisioning of infrastructure without needing to wait several days or weeks for an access request to be approved and provisioned.
This also reduces the burden on the accounting team that processes expense reports for team members each month. Each team member’s account is now part of consolidated billing.
Over the years, our non-production infrastructure resources have grown organically without accountability, cost controls, naming conventions, provisioning automation, or security best practices. This includes resources created in the GCP gitlab-internal project, AWS gitlab-np account, and DigitalOcean using dev-resources.
Epic 257 was created to iterate on our processes. In FY21-Q3, we created company-wide infrastructure standards which solved the “naming things is hard” problem with labels, tags, and naming conventions in our AWS and GCP organization accounts. The infrastructure standards define realms to create separate security boundary namespaces for different use cases. For our sandbox use cases, we’ve created a sandbox realm for individual users and department realms for shared collaboration projects, notably the Engineering Development realm which allows each of the department groups (functional teams) to have a shared AWS account or GCP project for creating infrastructure.
Jeff Martin developed the first release of HackyStack that powers the GitLab Sandbox Cloud. We developed the tooling in-house since the existing industry tools only solve 1-3 of the Technical Problems We're Solving and we wanted to automate the workflow end-to-end.
We are developing HackyStack as an open source project to allow other infrastructure or IT team to simplify their processes for provisioning sandbox accounts. HackyStack is not designed for individual use (yet). As we evolve, we'll be able to advocate HackyStack to partners and customers for deploying demo, testing, or training infrastructure without long manual provisioning documentation or burdening internal infrastructure team members.
This project was built using Laravel instead of other viable languages due to Jeff's prior experience and proficiency with Laravel to achieve the most efficient time to business value. This builds on the success of the GitLab Demo Systems that is powered by the demosys-portal.
For those who are not familiar with Laravel, it is the PHP equivalent of Ruby on Rails and Django and has seen tremendous community popularity in recent years since PHP has made revolutionary improvements in recent years with PHP 5.x and PHP 7.x. This project also allows us to dogfood GitLab CI/CD capabilities for PHP projects.
See the issue trackers for the latest up-to-date information.
dev-resources and support-resources.Phase 4 - Automated provisioning of AWS accounts and GCP projects for each user and team with streamlined/automated access requests (aka “Automate the manufacturing of everyone’s green LEGO board”). This is being achieved with the HackyStack open source project that Jeff is building.
Phase 4.5 - Migrate everyone’s resources in shared accounts into respective isolated accounts and apply labels/tags for cost management and reporting. See it-infra#86 Project Playground for details.
Phase 5 - Curate centralized library of Terraform modules, Ansible roles, Packer images, Docker images, and other scripts that have best practice security standards are used for deploying common infrastructure (aka “Provide everyone a box of LEGO bricks and the tools to deploy them”). Integrate GitLab Environment Toolkit for deploying GitLab in decentralized test environments (user sandboxes, community member environments, etc). This will be open source with the community so partners and customer POCs can take advantage of what we have. This will solve Sid’s request to ensuring we’re all on the same page and using the same library for the millions of GitLab users.
Phase 6 - Create “easy button” for deploying the library of infrastructure (aka the LEGO kits) into a topology builder.
Product and Revenue Enablement - Since a lot of provisioning functionality uses GitLab CI/CD and GitOps, we are dogfooding the GitLab product and allows users to manage their infrastructure-as-code in a GitLab repository and extend capabilities with other GitLab features as they see fit.
HackyStack and GitLab Premium Features - We will eventually add premium features to HackyStack that would require features included with a GitLab paid subscription.
Please post your ideas on Slack or in the issue tracker so we can discuss the best ways to implement them.
The GitLab Sandbox Cloud is a part-time side project for Jeff Martin, so we are not able to commit to firm timelines for feature development. We will try to resolve bugs promptly, however please comment in an issue and tag @jeffersonmartin if there is a feature or capability that needs to be prioritized.
Please direct any questions about the Sandbox Cloud or HackyStack to Jeff Martin or Dave Smith.