How GitLab automates engineering management

Nov 16, 2021 · 11 min read · Leave a comment
Seth Berger GitLab profile

As an engineer, figuring out how to automate your work becomes an important aspect of your job. From writing powerful dotfiles, to customizing bash scripts, to writing robust and rigorous tests, engineers regularly look for ways to automate their repetitive work.

At GitLab, engineering managers are no different and are constantly looking for ways to automate their work. I asked engineering managers at GitLab to share their automation scripts and their responses were overflowing.

From automating their 1:1 document creation, to integrating GitLab with Google Sheets, to writing utilities to provide executive summaries, GitLab team members take advantage of the rich API that GitLab provides to organize the mountains of information that they sort through on a regular basis.

For this blog post, I’m sharing a repo that contains just a few of the many scripts that our team members use. These scripts were originally written by engineering manager Rachel Nienaber. Rachel’s Infrastructure team is tasked with the exciting work of coordinating large scale infrastructure and code improvements. The work involves coordinating and sequencing lots of issues and epics, and ensuring the work gets done at just the right time and in the right order. Because of the breadth and scale of the work, she has created a handful of scripts that parse issues and epics in order to gain better visibility into the work that needs to be done.

In the repo, there are three scripts. I’ll provide a quick overview of the first two, and then dive into the code on the last one.

Issues not in epics

Since the Infrastructure team leans on epics to organize their issues, they also want to be able to organize work that may not be part of an epic. The issues_not_in_epics.rb script iterates through issues not in an epic and updates the description of a single hard-coded issue with a table summarizing those issues. The script is run on a daily basis via a scheduled pipeline. This ensures that issues do not slip through the cracks.

Epic summary

This script, epic_summary.rb, was written to solve the problem of having to look in multiple places to understand the status of each project. By grouping all status information into one place it’s easy to see what the team is working on, and what projects will be coming up next.

As input it takes a designated epic ID and updates the description of that epic by crawling sub-epics and extracting the following data from those epics:

You can see an example of the output from the script on this epic.

Part of what makes this script simple is that the Infrastructure team always updates the bottom of all their epic descriptions with the following markdown.

## Status {DATE}
{commentary of the status}

By consistently using that very simple markdown, the following snippet of code can reliably extract the status for each epic:

 if description!= nil && description.index("## Status")

    end_location = description.length

    if description.index("mermaid")
      end_location = description.index("mermaid")-6
    end

    status = description[description.index("## Status")+10..end_location]
  end

The code above certainly won’t win any algorithm challenges, but that’s kind of the point and what we aim to do with boring solutions.

You’ll notice the code above adjusts what is parsed to exclude a mermaid diagram that might appear after the ## Status markdown. That diagram gets maintained with the epic_issue_relationship.rb script.

Epic issue relationship

This script updates either a specific epic or all epics, depending on the command line option, with a mermaid diagram that shows the relationship between issues and the order that those issues need to be completed by examining how they are related to one another. Adding a mermaid diagram to the description was introduced by Sean McGivern, a staff engineer on the Scalability team. It creates brilliant diagrams like this one from this epic.

Mermaid Diagram

Let’s walk through the code.

The script uses the Docopt gem to parse and accept several input parameters.

options = Docopt::docopt(docstring)
token = options.fetch('--token')
group_id = options.fetch('--groupid')
epic_id = options.fetch('--epicid', nil)
dry_run = options.fetch('--dry-run', false)

Then a connection to the GitLab instance is created, taking advantage of the GitLab gem which is extended in lib/gitlab_client/epics.rb to include a few extra methods.

Gitlab.configure do |config|
  config.endpoint = 'https://gitlab.com/api/v4'
  config.private_token = token
end

If an epic id is passed in, then the update_mermaid will run only for a specific epic. Otherwise, the code searches for epics that match the two labels, workflow-infra::In Progress and team::Scalability and are also opened. Only when the matching epics do not have child epics, is update_mermaid run.

if epic_id
  update_mermaid(token: token, group_id: group_id, epic_id: epic_id, dry_run: dry_run)
else
  Gitlab.epics(group_id, 'workflow-infra::In Progress,team::Scalability', options: { state: 'opened' }).each do |epic|
    if Gitlab.epic_epics(epic['group_id'], epic['iid']).count == 0
      update_mermaid(token: token, group_id: group_id, epic_id: epic['iid'], dry_run: dry_run)
    end
  end
end

Finally the most exciting part of the script is the method update_mermaid method.

Below the code sets up variables, and looks to see if a mermaid diagram exists in the epic description that it should populate. Note, that if a mermaid diagram does not exist in the epic already, this script will not create one. Each epic should already have a mermaid diagram placeholder inserted after the status header.

def update_mermaid(token:, group_id:, epic_id:, dry_run:)
  in_epic = Set.new
  from_relations = Set.new
  relations = Set.new
  mermaid = ['graph TD']
  original_description = Gitlab.epic(group_id, epic_id).description

  unless original_description =~ MERMAID_REGEX
    puts "#{epic_id} does not have a Mermaid diagram"
    return
  end

Next the code iterates through each of the issues in the epic and assigns a graph_id for each issue that will be part of the mermaid diagram. It also adds the key_fields to the in_epic Set. The code assigns title along with an emoji so that the mermaid diagram is visually richer. After that the graph nodes are added to the mermaid diagram.

 Gitlab.epic_issues(group_id, epic_id).each do |issue|
    iid = issue['iid']
    graph_id = id(issue)

    in_epic << key_fields(issue)

    title = "##{iid}"
    title = "🎯 #{title}" if issue['labels'].include?('exit criterion')
    if issue['state'] == 'closed'
      title = "✅ #{title}"
    elsif issue['assignees'].any?
      title = "⏳ #{title}"
    end

    mermaid << "  #{graph_id}[\"#{title}\"]"
    mermaid << "  click #{graph_id} \"#{issue['web_url']}\" \"#{issue['title'].gsub('"', "'")}\""

After adding the graph nodes above, the code iterates through the links associated with each issue. The code determines if the issue is blocked by or blocks another issue. Knowing the direction of this relationship defines which direction the arrow in the mermaid diagram should point.

The code also adds both the issue and link to the from_relations set, which will automatically deduplicate entries.

    Gitlab.issue_links(issue['project_id'], issue['iid']).each do |link|
      case link['link_type']
      when 'is_blocked_by'
        source = id(link)
        destination = graph_id
      when 'blocks'
        source = graph_id
        destination = id(link)
      else
        next
      end

      from_relations << key_fields(issue)
      from_relations << key_fields(link)

      unless relations.include?([source, destination])
        mermaid << "  #{source} --> #{destination}"
        relations << [source, destination]
      end
    end

Finally, the code looks at the “extra” issues, which are issues that are not directly part of the epic, but are related to issues in the epic. These are the most important issues to ensure are on the diagram, since they represent issue dependencies that are outside the epic and would otherwise not show up when viewing an epic page in GitLab.

The code then updates the epic description by calling the GitLab API and setting the new description.

  (from_relations - in_epic).each do |extra_issue|
    mermaid << "  #{id(extra_issue)}[\"❌ ##{extra_issue['iid']}\"]"
    mermaid << "  click #{id(extra_issue)} \"#{extra_issue['web_url']}\" \"#{extra_issue['title'].gsub('"', "'")}\""
  end

  mermaid_string = mermaid.join("\n")
  new_description = original_description
                        .gsub(MERMAID_REGEX,
                              "\n\\1\n```mermaid\n#{mermaid_string}\n```\n")

    Gitlab.edit_epic(group_id, epic_id, description: new_description)
end

The above scripts help engineering managers efficiently know about all the issues their team members are working on, the status of their team’s epics and how all the work fits together.

The scripts only rely on team members doing two things manually:

The scripts can be run as part of a regular scheduled pipeline. With the reports generated on a scheduled basis, engineering managers can regularly get summarized information that helps make them and their teams more productive.

“Here's how @gitlab automates engineering management, with scripts your team can use too” – Seth Berger

Click to tweet

Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license