GitLab 12.7 released with Parent-Child Pipelines and Windows Shared Runners Beta
GitLab 12.7 released with Parent-Child Pipelines, Windows Shared Runners Beta, Pipeline Resource Groups, and much more!
GitLab 12.7 is now available with improvements to make your pipelines and teams more efficient and effective. Automation and pipelines are foundational for effective DevOps teams, and in 12.7 we’re delivering multiple improvements to make you faster and more efficient, including Parent-Child Pipelines, Pipeline Resource Groups, and Windows Shared Runners in beta on GitLab.com.
Large, complex pipelines can be both slow to execute and hard to understand. Parent-child pipelines will speed things up by allowing separate, child pipelines to run concurrently. By also separating and simplifying visualization and configuration per pipeline, and even allowing shared, reusable configuration, parent-child pipelines are easier to manage and will make both your CI/CD and your team more efficient.
Manage Shared Pipeline Resources
Many organizations have shared environments and resources where they want to prevent multiple changes from being made at the same time. Resource Groups help you limit pipeline concurrency to more efficiently and effectively manage jobs and resources.
Windows Shared Runners Beta
Windows developers can now choose to take advantage of Shared Runners on GitLab.com, instead of or in addition to setting up their own. This beta release enables running CI/CD jobs on Windows virtual machines with the added efficiency of a fully-managed, auto-scaling, and secure environment, managed by the GitLab.com team.
Faster Merge Request Management
Timely code review keeps your team shipping. Code Review Analytics will make it easy to find Merge Requests in review that need intervention, helping teams manage their cycle time. Improved Merge Request widgets now show when changes have made it to a specific environment, saving time tracking down the latest deploy.
Complex applications often require complex pipelines, which can become slow and hard to understand.
As complexity in pipelines increases, visualizations grow unwieldy,
configuration more convoluted, and execution slows down.
When this happens, splitting complex pipelines into multiple pipelines, organized
in a parent-children relationship, can improve performance and make pipelines easier to think about:
performance can be improved because child pipelines can run
concurrently, while configuration and visualization can be compartmentalized into different
files or views.
In GitLab 12.7 you can now define these separate pipelines using separate YAML files.
The .gitlab-ci.yml remains the primary entry point, but from there you can trigger
any other YAML file in the repo as its own child pipeline with attribution back to the
parent. We also support using includes to facilitate code reuse between these different
pipelines. This can be really great for monorepos, for example, where the
configuration for each component in the repository can be developed and stored
separately, and all the child pipelines can run concurrently — even meeting back
at the end for shared bundling and deployment.
We are happy to announce the availability of Windows Shared
Runners hosted by GitLab, now in
beta. You can
take advantage of a fully-managed, auto-scaling, and secure environment
for running your CI/CD jobs on Windows virtual machines, hosted on
the same GCP infrastructure as GitLab.com. Windows Shared Runners are
pre-configured with various software packages such as the Chocolately
package manager for Windows, Visual Studio 2019 Build Tools, and
Microsoft .Net Framework, to name a few.
As always, you can continue to install and manage Windows
Runners on your
own infrastructure, to use alongside or instead of the newly available
GitLab.com Runners managed by the GitLab team.
For more information on pricing, limitations, and configuration,
along with step-by-step instructions on how to get started, see the
Windows Runners beta launch
post.
Most often users want to parallelize their CI/CD as much as possible in
order to speed up the total running time. However, there are some use
cases when you may want to limit concurrency. For example, if you wanted
to prevent a job from a different pipeline executing at the same time
in the same environment. We see this often when physical hardware like
a smart phone, computer chip, or embedded IoT device is used to run
tests before releasing the software broadly. Trying to run tests from
multiple pipelines, or even multiple jobs in the same pipeline, and at the
same time can result in corrupted data, invalid test results, or
even bricking the hardware.
Now, with Resource Groups you can limit pipeline concurrency to force
jobs to execute sequentially, ensuring resources are only utilized as
intended. Using the resource_group key in gitlab-ci.yml you can
specify environments that are part of a Resource Group for each job that
you want to constrain. When the job requests a Runner and the resource
already has an existing job running, it will remain queued until the
existing job has completed. You can even define multiple Resource Groups
per job if, for example, you have multiple IoT devices that you test
with and you want a test job to run on any one device in the group. Another
great use case for this is when using Terraform for managing infrastructure as code, and you really want to be sure you’re only making one change to a given environment at a time.
The ability to limit pipeline concurrency has been a highly requested
feature and we are happy to make it available in this release. We look
forward to hearing your feedback, enhancement requests, and success
stories.
A recommended part of any software development process, code review
enables peers and automated processes to check a proposed change to a
repository.
It’s also where incoming changes can stall, because of a
missed handoff, a key individual going on vacation, or a backlog of
items to review. Cycle time depends on timely code review to keep your
team shipping.
In order to help GitLab instances stay on top of this key part of the
development process, we’re proud to introduce Code Review Analytics to
highlight the status of merge requests in review.
Code Review Analytics starts the review process as soon as a merge
request receives a non-author comment, and keeps track of how long a
merge request has been open since. The highlighted merge requests in the
table show review time in descending order, so it’s easy to find merge
requests that might need intervention or breaking down further.
The Merge Request widget now shows the environment name and the time of
the deployment in every merge request. Previously, in order to determine
when the last deploy took place, one would need to go through the list
of commits and pipelines to get this information. By tracking merge
requests, developers will be able to easily see when their changes made
it into a certain environment.
Groups can be used for organizing projects and groups of people. A
typical usage pattern is to create groups for different teams, e.g.
Engineering, Legal, and Operations, and share relevant projects with each
group.
For large instances with thousands of projects, sharing individual
projects with a group quickly becomes unscalable. To make this easier,
we’re introducing the ability to share a group with another group,
removing the need to share individual project links with a group.
Finding precisely the issues, merge requests, and epics you want can be
difficult, especially when you want to omit a subset of the results.
GitLab now supports excluding results while filtering issues, merge
requests, epics, and issues cards within an Issue Board using a not
or an “is not”
(!=) operator.
The Blame view allows you to trace the history of a file line by line,
and inspect each commit. In GitLab 12.7, you can easily follow the
history of a particular line, using the new button titled View blame
prior to this change.
With GitLab 12.7, you can now install a custom version of pip in
Dependency Scanning by using the
DS_PIP_VERSION variable.
When this is set, it will be passed down to the Dependency Scanning analyzers.
As a step further into GitLab
Releases, we make
your journey through an audit easier by automatically including events
for creating or editing Releases, downloading artifacts, and
associating or dissociating a milestone from within GitLab’s audit logs.
We recently launched the GitLab Conan Repository to support C/C++
developers in publishing and downloading their dependencies. However,
in order to leverage GitLab CI/CD, they were forced to use their
user name and password, which is inefficient and a potential security
risk.
In GitLab 12.7, we are excited to announce that users can now leverage
the predefined environment variable, CI_JOB_TOKEN, to authenticate to
their Conan Repository and easily publish and install their dependencies.
Administrators can now prevent users from changing their profile name to
increase traceability of user actions. This additional control over
identities will allow compliance-minded organizations the ability to
ensure GitLab supports their internal policies for audit logging and
identity verification.
This is an instance-level feature that requires administrative access and is currently available for self-managed customers only. You can follow and contribute to this issue discussing whether to enable this feature for GitLab.com users in a future release.
We have added support for an API that retrieves the list of merge
requests shipped within a given deployment. This is useful for tracking
when a merge request was merged to a certain environment.
When rolling out changes to your application, it is possible to release
production changes to only a portion of your Kubernetes pods as a risk
mitigation strategy. By releasing production changes gradually, error
rates or performance degradation can be monitored, and if there are no
problems, all pods can be updated. GitLab supports both manually
triggered and timed rollouts to a Kubernetes production system using
Incremental Rollouts, to support both Continuously Delivered and
Continuously Deployed applications. They are already available in Auto
DevOps projects. We have prepared a new guide that shows how to achieve
the same result when using only GitLab CI/CD.
Feature Flags allow you to enable or disable a recently introduced
feature in your production environment, to reduce the risk of breaking
your application until the feature is fully tested. Now you can
conveniently toggle flags on or off inside GitLab directly from the
Feature Flag list. Previously, feature flags could only be toggled using
the API.
Errors can be noisy and plentiful which makes triage processes
time-consuming, because it is difficult to sort through the cruft to
find the critical ones. Being able to ignore an error in the GitLab UI,
gives you the ability to clear out errors that don’t need attention, so
that you can easily focus on the ones that require fixing. Ignoring
non-critical errors makes the error list easy to scan and triage. The
Ignore button can be found on the specific detail page for the error
and on the error list.
Suggesting changes
in merge requests makes proposing improvements easy. However, if you
use a push rule to require a specific format for all commit messages,
in most cases it wasn’t possible to apply the suggested change, because
the commit message generated by GitLab wouldn’t match the validation
pattern for the push rule.
GitLab 12.7 now supports configuring a commit message template for the
commits created by GitLab when applying a suggested change, so to that
is valid according to your commit message push rule.
When browsing a project, looking at its dependencies is often useful,
but dependencies are typically stored in machine-generated files that
link to a public package registry.
Now, when viewing a Rust project’s Cargo.toml dependency file, GitLab will
detect and link packages to crates.io, so that it
is easier to understand its dependencies. Previously, in GitLab
9.3
support was added for Go, Ruby, Node.js, Python, PHP, and Objective-C.
With OpenSSH switching to SHA-256 by default
in 2015, displaying a MD5
hash for SSH keys is not useful. Thanks to a community contribution,
you’re now able to see the SHA-256 fingerprint for both SSH and deploy
keys - and query
for a user via the key’s fingerprint, thanks to the addition of
this new API.
Thank you to Roger Meier (@bufferoverflow) for the contribution!
GitLab comes out-of-the-box with a number of useful metrics dashboards for
monitoring your application’s health. In 12.7, you can now easily
duplicate one of those default dashboards in order to make slight
customizations to add, for example, your application’s specific system
or business metrics.
GitLab Pages now has a way to disable non-authorized access separate
from project privacy settings, which will then require all users to log
in prior to accessing the site even if the project is set to public.
This setting can be enabled by a GitLab administrator via a checkbox in the Admin Settings for Pages.
Triage of errors is challenging because they are noisy, and pertinent
information is difficult to find. The error detail page in GitLab
surfaces the most important attributes of an error. With GitLab 12.7,
those details now include language and urgency, intended to help
determine the root cause of the error as quickly as possible.
In GitLab 12.7 we’ve updated dependencies across our indexer and GitLab
to support Elasticsearch 7.x alongside our existing support for
Elasticsearch 6.x. This is the most requested
feature
for our Elasticsearch integration so we’re very excited to be including
this in the release.
With the update to support Elasticsearch 7.x we’re also shipping version
2.0.0 of our indexer which officially provides this support. As
previously
mentioned,
Elasticsearch 5.6.x is no longer supported for use with GitLab.
A new API route is available under /projects/:id/services that
provides a list of all active project services for that given project.
Previously, the GitLab API included routes that allowed for creating,
editing, and deleting services–but no route to get this list.
The ability to get a list of active services makes it easier for
developers to programmatically add and edit services on their projects
from the API.
Documentation was added to provide instructions for connecting GitLab to an external Minio instance for object storage.
The GitLab chart no longer manages the life cycle of the GitLab operator CRD (Kubernetes Custom Resource Definition). Installation of the CRD now can be done directly with kubectl. For new instructions on installing CRD, see the GitLab Operator installation docs.
The flag to disable gitaly has been moved to a global setting. This simplifies the process for disabling Gitaly so that you no longer need to edit multiple settings across the various services. For new instructions on how to disable Gitaly to leverage an external Gitaly service, see Configure this chart with External Gitaly.
We’re also releasing GitLab Runner 12.7 today! GitLab Runner is the
open source project that is used to run your CI/CD jobs and send the
results back to GitLab.
GitLab’s log
system allows
administrators to monitor and evaluate log files to better understand
the state of an instance. These logs have a wide variety of use cases,
including performance monitoring, analytics, and system auditing.
Some logs, however, are only offered in an unstructured format — making
them challenging to parse. One of these unstructured logs was
application.log, which recorded application activity including
project, group, and user events. In 12.7, we’ve introduced a more
flexible logging
system in
GitLab, and introduced a JSON-formatted version of application.log in
the form of application_json.log.
Creating a structured version of this logfile opens up a number of
interesting use cases, like ingesting into a monitoring tool for event
auditing, sending these events to a visualization tool to build
customized dashboards, and many more.
Incorporating tabular data into Markdown can be a tedious and labor
intensive process. As of 12.7, you can now copy content from a
spreadsheet and paste it directly into a Markdown editor within GitLab.
Automatically converting the spreadsheet into valid Markdown
syntax let’s you spend less time formatting and more time creating
amazing things.
The Web IDE in GitLab is a great tool for contributing changes to a
project. However, the lack of a persistent file system can become a
challenge. Situations may occur where users make changes to multiple
files, but not all changes are staged and committed prior to browsing
away from the Web IDE page. This could result in the loss of
those changes.
In order to make the Web IDE more accessible to users and prevent more
cases of accidental loss, changes in the Web IDE will now be
automatically staged. When a user clicks Commit, the following
commit screen will give them the ability to add their commit message and
select their branch instead of giving the option to stage changes.
Previously, deleting a pipeline was only possible from the API. As of
12.7 users with owner permissions to a project can click on the new
Delete button to delete a specific pipeline directly from the
Pipeline Details page. This non-reversible action expires the pipeline
caches and deletes all related objects (builds, logs, artifacts, and
triggers).
A key benefit of being able to delete a pipeline from the UI is the
ability to take immediate actions to protect leaked secrets if a job in
the pipeline exposes private keys in plain text. And an even more commom
use case for deleting pipelines is the need to clean-up a messy CI history
littered with failed pipelines resulting from CI configuration attempts or
experiments; a side benefit of cleanup being assurance that undesirable
pipelines are not inadvertently used again.
When uploading a large image, like a widescreen website layout, to the Designs tab in an issue, there
was difficulty in seeing the detail of the image because it was
displayed in the fixed-width viewport. We now include the ability to
zoom in on the Design so you can get into the details! You’ll find the
zoom controls at the bottom of the image.
Auto DevOps is a great way to get started with modern DevOps practices
for any project. However, until now, Auto DevOps needed to run a CI
pipeline to determine compatibility with a project by checking if an
existing Dockerfile or a matching buildpack exists for the project.
In an effort to make it more efficient and taking advantage of the new
workflow:rules
feature in GitLab CI, Auto DevOps will only run if the project contains
a Dockerfile or if a matching buildpack exists for the project’s
language.
Installing Kubernetes applications using GitLab
CI
provides a great way to customize Helm charts and custom resources
(CRDs) prior to installation. As part of this release we have added
templates for installing
Cert-Manager
as well as
GitLab Runner
using GitLab CI. Installing the GitLab Runner helm chart via GitLab CI
allows users to configure settings they previously could not, such as number of
concurrent jobs or the jobs check interval.
Service Desk email responses can now be customized per your
organization! Simply add template Markdown files to your repository and
when the Service Desk responds to a user, the templates are used! This
will allow custom branding and messaging to provide an optimal
experience for customers.
GitLab Design
Management
allows users to upload design assets (e.g. mockups) to GitLab issues and
store them in one place.
GitLab Geo now supports replicating these Design Management assets to
secondary nodes, ensuring that distributed teams can access them from
the closest Geo node. Because Design Management assets are replicated,
they can also be restored from a secondary node.
The Service Desk allows new issues to be created by sending an email to
a unique address. When these new Service Desk issues are created, it would be beneficial if they
could be automatically assigned to a specific user, given a label or
assigned to a milestone. You can now do that by creating a template to
be included with all new Service Desk issues. Include any of the quick
actions in the
template and they will activate when the issue is created.
You’re now able to adjust appearance settings of your instance -
including attributes like your instance’s title, description, favicon,
logo, and others - through a new API.
Thank you to Fabio Huser and Siemens for the
contribution!
Digging through a stack trace is cumbersome enough without also having
to determine what version impacted the affected file. In GitLab 12.7,
the commit that most likely caused the error is automatically surfaced
on the error detail page. Being able to automatically associate the
commit with the suspect error you are seeing can significantly reduce
the time to resolve the error. This gives you the ability to immediately
roll back the change, or fix it with a patch.
Please note that in order to take advantage of this feature, you will
need to name your Sentry Release objects with the SHA of the deployed
commit.
Proposing improvements to a Merge Request using a Suggested Change
makes collaborating easier by applying the change and resolving the
discussion with a single click.
In GitLab 12.7, improved diff highlighting of suggested changes makes
it obvious which words and characters have changed, so that you can
apply the suggestion with confidence.
Every team has unique needs. Oftentimes, it’s necessary to keep an
Issue open beyond the life-cycle of a single Merge Request or to
reference an Issue in a commit without the intent of closing the Issue.
Prior to this release, teams did not have a way to disable the default
behavior of automatically closing an Issue by mentioning it in a Merge
Request or commit. As a step towards providing teams with more granular
control over this functionality, you can now disable automatic closing
of Issues within your Project’s settings.
Once you’ve identified the root cause of an error, deployed the fix, and
verified its success (all from within GitLab). With GitLab 12.7, it is
now possible to resolve the error without switching to Sentry, just with
a click of a button. The Resolve button can be found on the specific
detail page for the error.
In GitLab 12.7, we have made stark improvements to the first page
response time of the /projects API. Previously, for some parameter
combinations, we were seeing response times as high as 30 seconds on
GitLab.com. With these changes, the majority of responses will be within
one second. Note that these improvements apply regardless of the
pagination strategy used.
We have introduced a new paginaton mechanism for the /projects API
endpoint. Previously, offset-based
pagination was
the only method available, which while providing flexible sorting and
filtering options, performs increasingly poorly for each page requested.
This poor performance characteristic put increasing loads on the GitLab
server, and also exhibited longer and longer response times.
With GitLab 12.7, we are introducing keyset-based
pagination.
While this method only allows for sorting based on project id, it
performs significantly faster with consistently low response times
regardless of which page you are requesting. Utilization of keyset
pagination for queries which retrieve many pages, will both reduce the
load on the GitLab server and result in faster data retrieval.
In 13.0, we will implement a configurable page depth
limit for
offset-based pagination, defaulting to a maximum depth of 50,000
records. This limit of 50,000 will be enforced on GitLab.com in 13.0.
It was already possible to skip the creation of a CI pipeline when using
ci skip (or skip ci) in your commit
message or by using
push
options,
but it was not possible to skip CI when rebasing. As of 12.7 it is now
possible to do so when using the rebase API.
GitLab Chart 3.0, released along with GitLab 12.7, is a new major
version of the GitLab Helm chart. Due to the significant changes in this
version, upgrading requires additional steps to be performed and should
be done in accordance with the upgrade
documentation. GitLab
Chart 3.0 includes functional improvements and updates to a number of
components, each of which are outlined below and linked to the GitLab
Chart 3.0 epic.
The GitLab Chart uses a fork of the nginx-ingress chart. GitLab Chart 3.0 pulls in changes that were made in the upstream nginx-ingress chart to ensure GitLab compatibility with Helm 2.15.0 and Helm 3.
The extensions/* and apps/beta* API versions stopped being supported in Kubernetes 1.16. Multiple upstream charts used by GitLab have been updated to stop using these API versions. GitLab chart 3.0 includes updated upstream charts: Prometheus chart 9.4.x, PostgreSQL chart 7.7.0, and Redis chart 10.3.x (no longer forked).
Sidekiq deployments now use unique selectors to avoid confusion over which deployments own a set of sidekiq pods when multiple deployments are created. For important information about upgrading Sidekiq deployments, refer to the upgrade documentation.
The version of Redis bundled in Omnibus GitLab has been updated from Redis 3.2.12 to Redis 5.0.7, which brings GitLab up to date with the latest stable release of Redis. Redis 5 includes a number of improvements. For more information on the changes made in Redis 5, see the Redis release announcement. A manual restart of Redis is required after upgrading. See the upgrade notes for instructions, and additional details for Redis HA.
Upgrades between certain versions can sometimes fail if background migrations are still running when the upgrade is attempted. Documentation has been added to Updating GitLab that explains how to check whether background migrations are still running.
The Ruby version included in GitLab has been updated from 2.6.3 to 2.6.5 to include some fixes and security patches.
Documentation was added on how to use the EXTERNAL_URL environment variable to make it easier to get a GitLab instance up and running.
Planned removal of PostgreSQL 9.6 and 10.x in GitLab 13.0
To take advantage of improved performance and functionality in PostgreSQL
11 such as
partitioning,
we plan to require PostgreSQL versions
11 and
12 in GitLab 13.x. To
accomplish this, we will be introducing support for PostgreSQL 11 in an
upcoming release of GitLab 12.x while maintaining support for 9.6 and
10.x. With the arrival of GitLab 13.0, PostgreSQL 11 will be required.
By minimally requiring PostgreSQL 11, we are able to leverage the new
features introduced, without the overhead of maintaining multiple code
paths. Going forward, we will be maintaining a yearly cadence of
PostgreSQL upgrades, with support for the second and third most recent
versions as soon as we are able to add them. We welcome your feedback on
the proposed removals. Please comment in the Move to PG 11 and
12 epic.
Offset pagination limit of 50,000 for /projects endpoint
An offset-based pagination limit of 50,000 will be applied on
GitLab.com, and by default on self-managed instances, to the /projects
API endpoint in GitLab 13.0. Integrations which make API calls with
offsets above 50,000 will need to switch to a keyset-based pagination
method, which
will offer significantly improved response times and reduced load on the
GitLab server. Self-managed instances will be able to customize the limit
to a desired value.
To provide the optimized performance, keyset-based pagination only offers
ordering based on project id. Use cases which require more flexible
ordering options can continue to use offset-based pagination, provided the
offsets remain below the limit. If use cases require flexible ordering
options with deep offsets, we recommend sorting client-side.
The Redis version packaged with Omnibus GitLab has been updated to Redis
5.0.7. When upgrading your instance of GitLab, you will need to restart
Redis after the upgrade so that the new version will be active. To restart
Redis, run sudo gitlab-ctl restart redis. If your instance has Redis HA
using
Sentinel,
you will need to follow specific upgrade steps that are documented in
Updating GitLab installed with the Omnibus GitLab
package
to avoid downtime.
GitLab 12.7 works towards implementing a more performant diff view. As part
of this work, we have changed how diffs are stored in the Redis cache. Users
upgrading their GitLab installation may see a temporary increase in Gitaly
load as the cache begins receiving traffic. If this severely impacts
availability, please disable the hset_redis_diff_caching feature flag.
This release of GitLab maps to a major release of the GitLab Chart, GitLab
Chart 3.0. If you have installed GitLab using the Helm Chart, you will need
to take some manual steps to upgrade from a previous version to GitLab Chart
3.0. For step-by-step instructions, refer to the upgrade
documentation.
Helm 2.15.x contains bugs that affect the use of the GitLab helm chart. Do
not use Helm 2.15.x with GitLab Chart. For supported versions of Helm, refer
to the install
documentation.
We want to hear from you
Enjoyed reading this blog post or have questions or feedback? Share your thoughts by creating a new topic in the GitLab community forum.
Share your feedback