GitLab Professional Services
Accelerate your software lifecycle with help from GitLab experts
Popular GitLab use cases
Enterprise Small Business Continuous Integration (CI/CD) Source Code Management (SCM) Out-of-the-box Pipelines (Auto DevOps) Security (DevSecOps) Agile Development Value Stream Management GitOpsGitLab Professional Services
Accelerate your software lifecycle with help from GitLab experts
Popular GitLab use cases
Enterprise Small Business Continuous Integration (CI/CD) Source Code Management (SCM) Out-of-the-box Pipelines (Auto DevOps) Security (DevSecOps) Agile Development Value Stream Management GitOpsAt times, users of GitLab.com can find that their IP address has been blocked due to rate limiting. Currently, rate limit parameters on GitLab.com are best described on the GitLab.com settings docs page. When this happens we may be able to determine what caused a block and relay that information back to the user.
A standard response is available as a macro: GitLab.com::Temp IP Ban
Please also see the log requests workflow for what information we can provide when responding.
Start by adding a positive filter on json.remote_ip
for the IP address provided by the user:
You can then drill down from there with positive and negative filters on fields to get the best results.
It can sometimes be unclear if a user has actually been blocked by our end or not. If they've been blocked by Rack Attack, we should be able to locate requests in Kibana that were blocked because of it.
To do so, enter the IP address into the main search field and set a positive filter on json.message
for Rack_Attack
.
You should see results similar to the following:
The existence of these results tells us that this user was blocked by Rack Attack and we can add the json.fullpath
field to see which exact path on GitLab.com each request tried to access.
Rack Attack can also throttle traffic. This is recognizable by the HTTP 429 response code. The preferred solution to this is to have the user make fewer requests. If that is not possible you can create an infrastructure issue with this template.
The following fields are the best to add to your search query in order to get the most important details on multiple requests at a glance.
json.status
- Outputs the HTTP status code that was returned for the request. We're usually looking for 401
(Unauthorized) and/or 403
(Forbidden).json.path
- The path on GitLab.com that was accessed by the request or the API endpoint that was hit.json.method
- Can be either GET
, POST
, PUT
, PATCH
, or DELETE
. The first three are the most common.json.env
- Can be blocked
, throttle
or track
. track
is used for diagnostics when changing rate limiting settings and does not affect users: from a support perspective, ignore track
. blocked
happens in reaction to too many failed authentication attempts, for example with automated Git HTTP traffic. throttle
means a user or IP is making too many requests per minute.These fields can be helpful but aren't essential.
json.controller
- Gives you a clue as to what part of GitLab.com was being accessed by a particular request.json.params
- Shows what user made the request, what action was taken, and on what resource it was taken on. This field shows what repository was targeted for requests to the container registry.json.matched
- This is the name of the Rack Attack rule used to limit this request (when json.env
is throttle
). This can help you find out which of the current limits the request surpassed.Numerous failed pushes or pulls to registry.gitlab.com
can result in an IP block.
You can list all log results for hits on the container registry by searching for the provided IP address and setting a positive filter on json.path
for /jwt/auth
.
json.status
json.path
json.params
json.controller
Failed pushes to the registry will always have JwtController
for the json.controller
field and /jwt/auth
for the json.path
field. Watch for :push,pull
in the json.params
field, indicating that the request is for a push.
A failed push will look like the following in Kibana.
Similar to a push, failed pulls from the registry will always have JwtController
for the json.controller
field and /jwt/auth
for the json.path
field. However, in json.params
only :pull
will be present.
A failed pull will look like the following in Kibana.
gitlab-ci-token
PullsUsers can also become blocked due to registry pulls from the user gitlab-ci-token
making (normal) unauthed requests to jwt/auth
. This user may be exempted from rate limiting in the future, it's being discussed in this issue.
An example request looks like:
To filter for these types of requests specifically, add the json.params
field.
Pushes or pulls to repositories containing LFS objects can result in an IP block if the user is unauthorized.
json.action
json.controller
json.method
json.status
Failed LFS pushes will always have upload_authorize
in the json.action
field, Projects::LfsStorageController
for the json.controller
field, and PUT
for json.method
.
A failed LFS push will look like the following in Kibana.
An IP can become blocked if a repository is cloned via HTTPS without authentication enough times.
The request will look like this:
json.status
: 401
json.controller
: Projects::GitHttpController
json.action
: info_refs
json.path
: /namespace/project.git/info/refs
json.ua
: If the request did not come from a browser this will reveal the operating system of the machine that attempted the clone along with the version of Git installed on it. Examples: git/2.11.1.windows.1
, git/2.17.2 (Apple Git-113)
, git/2.17.1
(Ubuntu 18.04 LTS), etc.
You should also see git-upload-pack
in the json.params
field.
An IP can become rate-limited if a customer attempts to export or download project exports too rapidly. See Project Import/Export
json.method
: Application_Rate_Limiter_Request
json.controller
: ProjectsController
json.action
: download_export
json.path
: /namespace/project/download_export