Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Identifying the Cause of IP Blocks on GitLab.com

On this page


At times, users of GitLab.com can find that their IP address has been blocked due to rate limiting. Currently, rate limit parameters on GitLab.com are best described on the Gitlab.com settings docs page. When this happens we may be able to determine what caused a block and relay that information back to the user.

Responding

A standard response is available as a macro: GitLab.com::Temp IP Ban

Please also see the log requests workflow for what information we can provide when responding.

Searching

Search Condition

Start by adding a positive filter on json.remote_ip for the IP address provided by the user:

Add remote_ip filter

You can then drill down from there with positive and negative filters on fields to get the best results.

Checking for Rack Attack Blocks

It can sometimes be unclear if a user has actually been blocked by our end or not. If they've been blocked by Rack Attack, we should be able to locate requests in Kibana that were blocked because of it.

To do so, enter the IP address into the main search field and set a positive filter on json.message for Rack_Attack.

Searching for an IP

You should see results similar to the following:

Checking Rack Attack results

The existence of these results tells us that this user was blocked by Rack Attack and we can add the json.fullpath field to see which exact path on GitLab.com each request tried to access.

Fields

Primary

The following fields are the best to add to your search query in order to get the most important details on multiple requests at a glance.

Secondary

These fields can be helpful but aren't essential.

Common Causes

Container Registry

Numerous failed pushes or pulls to registry.gitlab.com can result in an IP block.

You can list all log results for hits on the container registry by searching for the provided IP address and setting a positive filter on json.path for /jwt/auth.

Useful Fields

Failed Push and Pull Examples

Push

Failed pushes to the registry will always have JwtController for the json.controller field and /jwt/auth for the json.path field. Watch for :push,pull in the json.params field, indicating that the request is for a push.

A failed push will look like the following in Kibana.

Failed Push

Pull

Similar to a push, failed pulls from the registry will always have JwtController for the json.controller field and /jwt/auth for the json.path field. However, in json.params only :pull will be present.

A failed pull will look like the following in Kibana.

Failed Pull

gitlab-ci-token Pulls

Users can also become blocked due to registry pulls from the user gitlab-ci-token making (normal) unauthed requests to jwt/auth. This user may be exempted from rate limiting in the future, it's being discussed in this issue.

An example request looks like:

gitlab-ci-token registry pull request

To filter for these types of requests specifically, add the json.params field.

LFS

Pushes or pulls to repositories containing LFS objects can result in an IP block if the user is unauthorized.

Useful Fields

Examples

Push

Failed LFS pushes will always have upload_authorize in the json.action field, Projects::LfsStorageController for the json.controller field, and PUT for json.method.

A failed LFS push will look like the following in Kibana.

Failed LFS Push

Cloning Fails

An IP can become blocked if a repository is cloned via HTTPS without authentication enough times.

The request will look like this:

Log results for a failed clone

Useful Fields

You should also see git-upload-pack in the json.params field.