Published on: February 13, 2025
4 min read
When platform teams move their CI/CD to GitLab, migrating container images shouldn't be the bottleneck. Follow this step-by-step guide to automate the pipeline migration process.
"We need to migrate hundreds of container images from Amazon Elastic Container Registry (ECR) to GitLab. Can you help?" This question kept coming up in conversations with platform engineers. They were modernizing their DevSecOps toolchain with GitLab but got stuck when faced with moving their container images. While each image transfer is simple, the sheer volume made it daunting.
One platform engineer perfectly said, "I know exactly what needs to be done – pull, retag, push. But I have 200 microservices, each with multiple tags. I can't justify spending weeks on this migration when I have critical infrastructure work."
That conversation sparked an idea. What if we could automate the entire process? When platform teams move their CI/CD to GitLab, migrating container images shouldn't be the bottleneck. The manual process is straightforward but repetitive – pull each image, retag it, and push it to GitLab's Container Registry. Multiply this by dozens of repositories and multiple tags per image, and you're looking at days or weeks of tedious work.
We set out to create a GitLab pipeline that would automatically do all this heavy lifting. The goal was simple: Give platform engineers a tool they could set up in minutes and let run overnight, waking up to find all their images migrated successfully.
First things first – security. We wanted to ensure teams could run this migration with minimal AWS permissions. Here's the read-only identity and access management (IAM) policy you'll need:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage"
],
"Resource": "*"
}
]
}
With security handled, the next step is setting up GitLab. We kept this minimal - you'll need to configure these variables in your CI/CD settings:
AWS_ACCOUNT_ID: Your AWS account number
AWS_DEFAULT_REGION: Your ECR region
AWS_ACCESS_KEY_ID: [Masked]
AWS_SECRET_ACCESS_KEY: [Masked]
BULK_MIGRATE: true
Now for the interesting part. We built the pipeline using Docker-in-Docker to handle all the image operations reliably:
image: docker:20.10
services:
- docker:20.10-dind
before_script:
- apk add --no-cache aws-cli jq
- aws sts get-caller-identity
- aws ecr get-login-password | docker login --username AWS --password-stdin
- docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}
The pipeline works in three phases, each building on the last:
First, it finds all your repositories:
REPOS=$(aws ecr describe-repositories --query 'repositories[*].repositoryName' --output text)
Then, for each repository, it gets all the tags:
TAGS=$(aws ecr describe-images --repository-name $repo --query 'imageDetails[*].imageTags[]' --output text)
Finally, it handles the actual migration:
docker pull ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${repo}:${tag}
docker tag ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${repo}:${tag} ${CI_REGISTRY_IMAGE}/${repo}:${tag}
docker push ${CI_REGISTRY_IMAGE}/${repo}:${tag}
Remember that platform engineer who didn't want to spend weeks on migration? Here's what this solution delivers:
Instead of writing scripts and babysitting the migration, the platform engineer could focus on more valuable work.
Getting started is straightforward:
.gitlab-ci.yml
to your repository.BULK_MIGRATE
to "true" to start the migration.Through helping teams with their migrations, we've learned a few things:
We've open-sourced this pipeline in our public GitLab repository because we believe platform engineers should spend time building valuable infrastructure, not copying container images. Feel free to adapt it for your needs or ask questions about implementation.
Get started with this and other package components with our CI/CD Catalog documentation.