Nov 18, 2020 - Michael Friedrich    

How to make Docker Hub rate limit monitoring a breeze

Docker Hub Rate Limits are enforced and we need to find ways to monitor the remaining pull requests. Explore some ways to create a monitoring plugin for Nagios/Icinga/Sensu/Zabbix and test-drive a new Prometheus exporter in combination with Grafana.

When we learned about the Docker Hub Rate Limit, we thought about ways to mitigate and analyse the new situation. Container images are widely used and adopted for sandbox environments in CI/CD pipelines and cloud-native production environments with app deployment in Kubernetes clusters.

Each docker pull request toward the central hub.docker.com container registry is being counted. When a defined limit is reached, future requests are blocked and might be delayed into the next free window. CI/CD jobs cannot be executed anymore after receiving a HTTP error 429 - too many requests and similar errors will be seen in production deployment logs for Kubernetes.

Docker defines this limit with 100 anonymous requests every six hours for the client's source IP address. If you have multiple container deployments behind an IP address, for example a company DMZ using a NAT, this limit can be reached very fast. A similar problem happens with watchtower tools which try to keep your container images updated, for example on your self-hosted GitLab Runner. The limit can be raised by logging in, and by getting a paid subscription.

The question is: Where can you see the current limit and the remaining pull requests?

How to check the request limit

The Docker documentation suggests to use CLI commands which invoke curl HTTP requests against the Docker Hub registry and parse the JSON response with jq.

Define the IMAGE variable once for the following CLI commands to use:

$ IMAGE="ratelimitpreview/test"

Obtain a token for authorization. Optionally print the variable value to verify its content.

$ TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" | jq -r .token)

$ echo $TOKEN

The next step is to simulate a docker pull request. Instead of using GET as HTTP request method, a HEAD request is sent which does not count toward the rate limit. The response headers contain the keys RateLimit-Limit and RateLimit-Remaining.

$ curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/$IMAGE/manifests/latest

The limit in the example is 2500 with remaining 2495 pull requests. 21600 defines the limit time window as six hours.

RateLimit-Limit: 2500;w=21600
RateLimit-Remaining: 2495;w=21600

RateLimit-Reset can be returned too, this will be the remaining time until the limits are reset.

Create a monitoring script

The CLI commands can be turned into a programming language of your choice which provides methods for HTTP requests and better response parsing. The algorithm needs to follow these steps:

  • Obtain an authorization token from Docker Hub. Username/password credentials can be optionally provided, otherwise the request happens anonymously.
  • Send a HEAD request to the Docker Hub registry and simulate a docker pull request
  • Parse the response headers and extract the values for RateLimit-Limit and RateLimit-Remaining
  • Print a summary of the received values

A plugin script which can be used by Nagios/Icinga/Sensu/Zabbix and others has additional requirements. It needs to implement the Monitoring Plugins API specification:

  • Print the limit and remaining count
  • Calculate a state: Ok, Warning, Critical, Unknown and print a helpful text on the shell
  • Add optional warning/critical thresholds for the remaining count. Whenever the count is lower than the threshold, the state changes to Warning/Critical and the exit code changes: OK=0, Warning=1, Critical=2, Unknown=3
  • Collect limit values as performance metrics for graphing and visualization
  • Add verbose mode and timeout parameters as plugin development best practices. If Docker Hub does not respond within 10 seconds as default, the plugin exits and returns Unknown as state.

You can download the check_docker_hub_limit.py plugin script and integrate it into your monitoring environment.

Use the monitoring plugin script

The check_docker_hub_limit.py plugin script plugin is written in Python 3 and requires the requests library. Follow the installation instructions and run the plugin script with the --help parameter to see all available options:

$ python check_docker_hub_limit.py --help

usage: check_docker_hub_limit.py [-h] [-w WARNING] [-c CRITICAL] [-v] [-t TIMEOUT]

Version: 2.0.0

optional arguments:
  -h, --help            show this help message and exit
  -w WARNING, --warning WARNING
                        warning threshold for remaining
  -c CRITICAL, --critical CRITICAL
                        critical threshold for remaining
  -v, --verbose         increase output verbosity
  -t TIMEOUT, --timeout TIMEOUT
                        Timeout in seconds (default 10s)

Run the script to fetch the current remaining count. The plugin script exit code returns 0 being OK.

$ python3 check_docker_hub_limit.py
OK - Docker Hub: Limit is 5000 remaining 4997|'limit'=5000 'remaining'=4997

$ echo $?
0

Specify the warning threshold with 10000 pulls, and the critical threshold with 3000. The example shows how the state changes to WARNING with a current count of 4999 remaining pull requests. The plugin script exit code changes to 1.

$ python3 check_docker_hub_limit.py -w 10000 -c 3000
WARNING - Docker Hub: Limit is 5000 remaining 4999|'limit'=5000 'remaining'=4999

$ echo $?
1

Specify a higher critical threshold with 5000. When the remaining count goes below this value, the plugin script returns CRITICAL and changes the exit state into 2.

$ python3 check_docker_hub_limit.py -w 10000 -c 5000
CRITICAL - Docker Hub: Limit is 5000 remaining 4998|'limit'=5000 'remaining'=4998

$ echo $?
2

When a timeout is reached, or another error is thrown, the exit state switches to 3 and the output state becomes UNKNOWN.

Use a Prometheus exporter for rate limit metrics

Prometheus scrapes metrics from HTTP endpoints. There is a variety of exporters for Prometheus to monitor host systems, HTTP endpoints, containers, databases, etc. Prometheus provides client libraries to make it easier to start writing your own custom exporter. The metrics need to be exported in a defined format.

The Docker Hub limit values can be fetched with obtaining an authorization token first, and then sending a HEAD request shown above. The code algorithm follows the ideas of the monitoring plugin. Instead of printing the values onto the shell, the metric values are exposed with an HTTP server. The Prometheus client libraries provide this functionality built-in.

We have created a Prometheus Exporter for Docker Hub Rate Limits using the Python client library. The repository provides a demo environment with docker-compose which starts the exporter, Prometheus and Grafana.

Ensure that docker-compose is installed and clone/download the repository. Then run the following commands:

$ cd example/docker-compose

$ docker-compose up -d

Navigate to http://localhost:3030 to access Grafana and explore the demo environment with the pre-built dashboard.

Grafana dashboard for Docker Hub Limit Prometheus Exporter

Grafana dashboard for Docker Hub Limits

More monitoring/observability ideas

Use the steps explained in this blog post to add Docker Hub limit monitoring. Evaluate the Prometheus exporter or the check plugin, or create your own monitoring scripts. Fork the repositories and send a MR our way!

The Prometheus exporter and the monitoring plugin script can help to see trends and calculate usage over time. Use your own local (GitLab) container registry or one of the available caching methods described in these blog posts:

Photo by Vidar Nordli-Mathisen from Unsplash.

A Cloud Native Transformation Learn how Ask Media Group modernized their architecture and development with microservices, containers, and Kubernetes. Learn more Arrow

Try all GitLab features - free for 30 days

GitLab is more than just source code management or CI/CD. It is a full software development lifecycle & DevOps tool in a single application.

Try GitLab Free
Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license

Try GitLab risk-free for 30 days.

No credit card required. Have questions? Contact us.

Gitlab x icon svg