Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Logging Working Group

On this page


Property Value
Date Created September 9, 2019
Date Ended February 1, 2020
Slack #wg_log-aggregation (only accessible from within the company)
Google Doc Logging Working Group (only accessible from within the company)
Issue Label WG-Logging (gitlab-com/-org)

Business Goal

  1. Monitor and Security teams will be responsible for triaging all logging issues
  2. Determine the long-term owner of the ultimate logging product and/or process
  3. Monitor and Security teams will be responsible for defining logging standards
  4. Clearly define how logging works at GitLab in the runbooks

Exit Criteria

  1. Triaging logging issues using the TriageBot
    • Security team will triage security and compliance issues to either the security team or a dev team
    • For dev teams, the Monitor will triage issues and assign to appropriate dev team.
    • Triagebot is modified by the security team to send slack notifications if higher priority logging issues aren't resolved in a specified timeframe.
  2. Monitor: APM group takes ownership of the overall logging products/process
    • Infrastructure will continue to own the system architecture and systems.
    • Monitor will slowly build out a comprehensive logging product to meet security and dev needs.
  3. Logging Standards
    • Monitor will own logging standards for dev teams
    • Security will own logging standards for compliance purposes
    • Monitor and Security team will both release logging standards
  4. Transfer all current state of logging to the logging (runbook)[]
    • Include all known details of the current system (DELKE remains 3rd-party-only for now)
    • Describes what logs are logged where
    • Describes current and proposed log standards
    • Details how to hook up new analyzers
    • Details about how TriageBot works
    • Details about who owns what parts of the triaging process


Security Automation is leading the logging effort at this time.

The DELKE project ticks a lot of the boxes that we wanted out of a logging system. To that end we are moving forward to make DELKE more DevOps ready and productizable.

We will move forward on Logging standards.

Other Investigations

What do other companies do?


LabKit is an application logging library Andrew Newdigate invented to help structure and standardize logging (similar to Graphite pings) throughout the Ruby and Go code bases

Where do logs go today?

Current logging system architecture Original Diagram

Noted issues

Roles and Responsibilities

Working Group Role Person Title
Facilitator Alex Groleau Security Software Engineer, Automation
Exec Sponsor Jan Urbanc Interim Director of Security
Member Stan Hu Engineering Fellow
Infrastructure Lead Andrew Newdigate Staff Engineer, Infrastructure
Member Ethan Urie Senior Backend Engineer, Security
Member Antony Saba Senior Threat Intelligence Engineer
Member Tomasz Mazukin Backend Engineer, Verify
Member Jayson Salazar Security Engineer, Security Operations
Member Paul Harrison Security Manager, Security Operations
Member Nik Sarosy Senior Security Analyst, Compliance

Requirements and Considerations






Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license