With GitLab 10.0, we shipped Auto DevOps for the Community and Enterprise
Editions. Read on for an in-depth look at our strategy behind it, and beyond.
I recently met with my colleagues
Courtland to give them the
lowdown on GitLab's DevOps vision: where we've come from and where we're headed.
You can watch the video of our discussion or check out the lightly edited
transcript below. You can also jump into the rabbit hole, starting with the meta
issue for GitLab DevOps.
CI/CD: Where we've come from
When I joined GitLab about a year ago, I created a vision document for
CI/CD, and outlined a lot of the
key things that I thought were missing in CI/CD in general, and going beyond CD.
I literally called one section "beyond CD" because I didn’t have a name for it
And in that document, I create an example pipeline to characterize all this
stuff, to show how the pieces fit together into a development lifecycle.
I love this diagram not only because it's complex and scary, but because when we
started, we had maybe four boxes filled in, and now we have 10 or 12 filled in. To
start with, we had code management and, obviously, builds and tests. And we kind
of did deployment, but not really.
Since then, we’ve added review apps – a specific example of deployments – which
is really awesome. We also added a more formalized mechanism for doing
deployments; actually recording deployments and deployment histories, keeping
track of environments, and everything else. Then we added Canary Deployments in
9.2 and code quality in 9.3. We added system monitoring with Prometheus in 9.0.
We don’t yet have what I called "business monitoring," which could mean
monitoring revenue, or clicks, or whatever you care about; but that’s coming. We
don't yet have load testing, but the Prometheus team is thinking about that.
We don't yet have a plan for feature flags, but I think it's a really important
And then we have this other dimension of pipelines, which is the relationship
between different codebases (or projects), and in 9.3 we introduced the first
version of multi-project pipelines.
So we've gone from a core view of three or four boxes to where 90 percent is
complete. That's pretty awesome.
It became obvious to me that we were viewing the scope with this hard line:
developer focused rather than an ops focused. For example, we’ll deploy into production,
and we might even watch the metrics related to your code in production, but
we’re not going to monitor your entire production app, because that’s
operations, and that’s clearly out of scope, right?
Where we're headed: Beyond CD
What hit me a few months ago is, "Why is that out of scope? That’s ridiculous.
No, we’re going to keep going. We're going to go past production into
operations." Most of this still applies, but instead of just monitoring the
system as it relates to a merge request, what about monitoring the system for
network errors, outages, or dependency problems? What if we don't stop at
production, and monitor things that are typically ops related that may not
involve a developer at all?
Then I realized that this thing I called Beyond CD, maybe it's really DevOps.
Maybe the whole thing is DevOps.
The DevOps tool chain
To offer some context: DevOps is hard to define, because everybody defines it
slightly differently. Sometimes DevOps is defined as the intersection of
development, operations, and quality assurance.
Image by Rajiv.Pant, derived from Devops.png:, CC BY 3.0
For the most part, my personal interest in DevOps has been in that intersection.
We do great code management; we’ve done that for quite a while. How do we get
that code into production? How do we get it into QA?
Review apps are a great example that fits squarely in that tiny, little triangle
in the middle of the Venn diagram. You take your code, you deploy it, which is
an operations thing, but you have it deployed in a temporary, ephemeral, app,
just for QA people (or designers, product managers, or anyone who is not a
primary coder), so they can test your application for quality assurance, feature
assurance, or whatever.
But now, I'm looking beyond the intersection. Here's the DevOps tool chain
definition from Wikipedia:
Image by Kharnagy (Own work) CC BY-SA 4.0, via Wikimedia Commons
Well, that’s everything! That’s not the intersection; that’s the union of
everything from code, to releasing, to monitoring. And that's where things get
confusing. Sometimes when people talk about DevOps, they’re not talking about
all of your code stuff. It’s the intersection parts that are the interesting
parts of DevOps. It’s the parts where we let developers get their code into
production easily. That slice, that intersection, of the Venn diagram, that’s
the interesting part about DevOps.
Having said that, as a product company, we are going to deliver things that are
pretty squarely on the development side, and, eventually, we’re going to deliver
things that are pretty squarely in the operations side. At some point, we may
have an operations dashboard that lets you understand your dependencies in your
network infrastructure, and your routers, and your whatever. That’s pretty far
fetched at this point, but it could happen. Why not? Just have GitLab be
your one operations dashboard, and then it’s not just about the intersection of
the DevOps, it’s the whole DevOps tool chain.
So, that is the whirlwind, high-level summary of where we've been, and a little
bit about where we’re going. Now let's get into specific issues.
The Ops Dashboard – #1788
We have a monitoring dashboard that's very developer centric. What about
taking that same content and slicing it from the operator's perspective? For a
moment, ignore all the stuff below, let’s just pretend there’s only the four
boxes at the top:
So an operator might want to know, "What’s the state of production?" If I'm a
developer I can go into a project, into environments, see the production
environment for that project, and I can see what the status is. But what if I
want to see all production environments? As an operations person, I care a
little less about individual projects than I care about "production." So this is
giving me the overview of "production." All of these little boxes would
represent production deploys of projects that you have in your GitLab
The view is explicitly convoluted because we had just introduced sub-groups and
I wanted to make sure this mechanism expanded. So ignore all the stuff below and
just look at the top-level dashboards. Or maybe one level down, which is already
still pretty complicated, but let’s say your marketing organization had
different properties than your other developer operations; you’d be able to see
really quickly what the status is. If something’s red, you’d be able to click
down, and see details.
You’d be able to see graphs like this, which are similar to what we already
provide, but from the other angle. As a developer I’m looking at the deploy, and
saying, "Oh, how did my deploy affect my performance?" But this is saying,
"How’s production? Is anything wrong with my entire production suite?"
This is really just scratching the surface of the ops views of things, but I
think it's going to become much more important as people embrace DevOps. You
want your developers to be talking the same language as your operations people.
In a lot of organizations, it’s already the same people – there are no separate
operations people. Developers push code to production, and they're paged if
something goes wrong. In others, developers and operators are separate, but they
want to work together towards DevOps.
Either way, you want to be using the same tools. You want to be able to point
to, for example, a memory bump that your operations people should also be able
to see. But if they’re using completely different tools, like New Relic and
Datadog, that kind of sucks. So let’s give them the same tools.
Pipeline view of environments – #28698
I particularly love this proposal, and I really want to see this happen soon.
The environments page today is just a list of environments showing the last
deployment. The picture tells you who deployed, which is good, and you can see
that the commit is from the same SHA as staging, which is kind of nice. I can
see the deploy board, and if there's a deploy ongoing, I’m able to see the state
as it rolls out. We don’t yet show you the current health of these pods; once
they're deployed, all we know is that they're deployed. This is how the
environment view is today, and it's centered around deployments.
Current Environment view
You can click through to see the deployment history and this is actually really
valuable because I can see who deployed things, how long ago, and if something
went wrong in production I can really quickly roll back and let the developers
have some space to go and figure out what went wrong.
Current Deployment History view
But this proposal turns it around to have more of a DevOps view of the thing.
Proposed pipeline view of Environments
The idea is to take the same application, and instead of just looking at a list
of environments, I’d be looking at columns with lots of review apps, and some
number of staging environments, and a production environment. Instead of just
showing you the SHA, we would show you, for example, what merge requests have
been merged into staging that are not yet in production. That’s a great
marriage of these two views, that you’d be able to see the diff between them.
This list, although it’s just a mockup, shows maybe the last five things that
were in production, or what was included in the last deploy, or whatever works
best for your environment. Showing what’s in the last deploy might be enough,
but for people who deploy 17 times a day, maybe that’s a little less useful, and
we just show history.
But then what about building in more of the operations kind of stuff, and
saying, "Alright, what’s the state of my pods?" Here we were flagging where the
error rate exceeded a threshold and there’s some alert that popped up. And here
we’re showing this automatic rollback kind of stuff, but basically just really
building on this ops view. Of course this is still a DevOps view, in the sense
that I’m looking at an individual project. So, one permutation of that would
marry that ops view of all of production. Or if I’m looking at a microservices
kind of thing, where there are five or 100 different projects, and I want to see
the status of all those really quickly. See
Dependency security – #28566
So, here, the idea is that you've deployed something in production, and some
module or something that you depend on has been updated, not by you, but by the
community, or someone else.
The easiest and most naive way to approach this is that with the next merge
request, or next CI/CD run, we would go and check to see if anything’s outdated.
And we might fail your CI/CD because of this.
It would make much more sense to run this stuff automatically. Even if, for
example, nobody pushes for seven days, and in the middle of that, there’s a
security release; just proactively run stuff and notify me. So, that's sort of a
second iteration of thinking about how you would notify somebody, and tell them,
"Oh, you’ve got a security change. You should go in and do something about it."
Now, the third iteration is, "Well, what would you do with that information?"
You’d go and maybe give it to your junior developer to go and make the change,
and point to the new version. And then, of course, you need to test that it
works. So, you’re going to create a merge request, and then test it, to make
sure that it still functions properly.
Well, why notify somebody, and tell the junior developer to go and do this? Why
don’t we just do it for you? Why don’t we just go and submit the merge request
for you, and then tell you what the results are. And, in fact, let’s go further,
and say, "Hey it passed. We just deployed into production for you." Why would
you have security vulnerability in place any longer than necessary?
And instead of having 100 alerts about 100 projects or microservices that all
need to get updated, you just get alerts about three of them that fail, that
actually have some weird dependency that it didn’t work on. And then, you can
focus on real problems.
So, that’s a glimpse at how we’re thinking about this.
This would definitely be an enterprise-level feature. And again, we've fleshed
out some ideas and it’s unscheduled, but it does really tie into the ops
Question: Enterprise Edition features
Courtland: You mentioned that sort of automation would be an enterprise edition
feature. Can you talk a little bit more about why a smaller development team,
like under 100 developers, wouldn’t get value out of something like that?
Mark: So, this is where things get a little tricky, because of course,
smaller developer teams would get value out of that too. Everybody would get
value out of that. Some of it has to do with proportionality. One test I like to
use is: is there some other way you could achieve the same thing, using
workarounds, and we’re just making it easier? And that’s a good case, here. You
can already do this, but we’re going to automate it. And automation is something
that affects larger companies a lot more, because they’ve got hundreds of
projects, with thousands of developers. And they just can’t deal with the scale,
or it’s worth dealing with the automation. Whereas, if you’ve got a small
developer, with a single project, you’re pretty much on top of it. And if
something changes, yeah, you just go ahead and fix it; you’re aware of it. The
bigger challenges are when you’re just not aware of how this thing might affect
one project that somebody’s almost forgotten about.
The other thing is that, just to be blunt, our concept that Enterprise Edition
is only for more than X people, is a little flawed. It’s that it
applies more to those companies, that those people value it more, and they’d be
willing to pay for it more, or however you judge your value there. Clearly,
small companies would value all this automation, and everything else, but
they’re not going to get as much incremental value out of it, as a larger
The other way to look at it is that this is pretty advanced stuff, and frankly, [Editor's note: Advancedness is not a criteria in open sourcing or not open sourcing. There are advanced features that are open source, such as Review Apps. There are basic features that are proprietary, such as File Locking. The criteria we use to decide which version the features go in are documented on our stewardship page.] Maybe there’d be levels to it,
it doesn’t deserve to be, free, open source. It’s probably really complicated
stuff, and you’re going to have to pay there.
right? There’d be a version that gives you an alert: we’ll run this test once a
day. Or even just have a blog post about how to do this: you set up a recurring,
scheduled pipeline job, once a day, to test if any of your dependencies have
been updated. And you can do that today and then it would alert you. But to
automate it, to actually, create a merge request for you, and everything else?
Well, that’s in the Enterprise feature. It’s not that version checking isn’t
important for everybody, but the automation around it really, really matters for
larger companies. Does that make sense?
Courtland: Yeah, I mean, I think that the first way you described it, in that,
"Yeah, everyone gets some value out a feature like this, but the overwhelming
value and use for this is in larger development teams," that resonated.
SLO and auto revert – #1661
This is a feature showing how we’re thinking about auto reverting something.
We’ve got canary deployments, and we have another feature we’re not currently
working on or scheduled, but it’s incremental rollout, so that you would not
just rollout to a single canary, or a bucket of canaries, but it would slowly
increment: 1 percent, then 5 percent, then 25 percent. But let’s say, at some point, during my
rollout, you detect an error.
This a mockup of what it would look like. You’re like, "Oh, error rates
increased by something above our threshold; let’s revert that one, go back, and
create a new issue, and alert somebody to take a look at it." Lately, I’m
thinking that I don’t know if I really want to automatically roll back, versus
just stop it in its canary form, and say, "Well, it’s canary. Let’s let canary
be there, so you can debug the canary, but just don’t let the canary go on
Error rate exceeding is a pretty tough one. But let’s say memory bumps up, and
you might be like, "Yeah, we added something, and it’s using more memory, and
we’re okay with that. Don’t stop my deploy just because it’s using more memory."
There might need to be human intervention in there, but somewhere along this
line we’re automating a lot of the deploy stuff.
Onboarding and adoption – #32638
Onboarding and adoption is a really big issue, with lots of different ideas for
how to improve onboarding, how to get people actually using idea to production,
improving auto deploy. Not a lot of visuals, so I won’t really talk about it,
but it’s definitely one of our top priorities; the next most important thing
we’re working on.
Cloud development – #32637
Cloud development is the idea that setting up your local host machine is
actually kind of a pain sometimes. Especially with microservices, where each
service can be in their own language, you don’t want to maintain Java, and Ruby,
and Node, and all these other versions of dependencies, and every time something
switches, you’ve got to reinstall a new version of stuff. Or even these days,
you might develop on an iPad, and you don’t have a local host to compile things.
Cloud9 is the biggest, well known thing, from an IDE perspective, and Amazon
bought them a little while ago. But even aside from the IDE portion of it, it’s
just being able to develop in the cloud, and being able to make some changes,
and then push them back; commit them to a repo.
We have a little bit of a demo like this, right now, with our web terminal. So,
if you have Kubernetes, you see this terminal button, and it just pops up the
terminal right in the staging server. And I can actually go ahead and edit a
file there, and... I just made a live change into my staging app.
Now, generally speaking, I would not actually recommend you do that, because
I’m messing with my staging app, that’s not what it's for. It makes an awesome
little demo, but it’s not what you should do. What we want to do is come up with
a way that people could do that, but have it be not on your staging app, but in
maybe a dev environment that is specifically for this purpose. But that also,
after you make your changes, and test them, and run them live, you can then go
and commit them back to version control, and close that loop. So there’s a whole
bunch of issues related to that. And to be honest, it was what we were hoping
that Koding would have provided for us, and we have an integration
with them, but it hasn’t worked out, really, the way that we had hoped. And so,
we’re looking at alternatives, and we think we can probably do this ourselves.
Anyway, that’s a big thing to flesh out.
GitLab PaaS – #32820
Heroku is awesome, because it gives you this really great platform that’s easy
to use, and gives you all this functionality on top of Amazon. Five or six years
ago it was super, brain-meltingly awesome to get people to do ops. For a
developer, I don’t have to be aware of how to do ops; Heroku just does ops
GitLab PaaS is basically the idea that you’ve got a lot of these components, and
we’re not going to invent them all from scratch. We’re going to rely on
Kubernetes, for example. But on top of Kubernetes, we could make an awesome
environment for ops. An ops environment, or a platform as a service. And so,
there’s an issue to discuss what it would take to do that. At some point in
time, this is a big item for us. If we can make it super really easy for you to
fully manage your ops environment via GitLab, and maybe, for example, never
touch the Kubernetes dashboard; never touch any of the tools, just use the
GitLab tools to do this. That’s pretty powerful.
Sort of related is an idea in the onboarding stuff, that on GitLab.com
we can actually provide you with a Kubernetes cluster; maybe a shared cluster. We
have to worry about security, of course. But imagine if you were a brand new
user on GitLab.com, and you push up an app, and you have nothing in there
specifically for GitLab, you just push up your code, and GitLab is like, "Oh,
that’s a Ruby app. Okay, I know how to build Ruby apps. Oh, and I also know how
to test Ruby apps. I’m just going to go and test them automatically for you."
And, "Oh, by the way, I know how to deploy this. I’m just going to go ahead and
deploy this to production." And we’ll make a
production.project-name.ephemeral-gitlabapps.com, whatever the hell, some domain
so that it’s not going to affect your actual production. But if you wanted to,
you would just point your DNS over to this production app, and you've got the
production app running on GitLab infrastructure. And that’s, really, what Heroku
But that also is an onboarding thing for us to make it really easy. Because if
we want everybody to have CI, well, let’s turn it on for you. That’s pretty
awesome. If we want everybody to have CD, we can’t just turn it on for you,
because you have to have a place to deploy it to. So, if we just provided you a
Kubernetes cluster ("everybody gets a cluster"), then you just got a place. And,
I mean, we’ll severely limit it. We’ll make it limited in some way, so that
you’re not going to run the production stuff for long there. Or if you do, you have
to pay for it. But we’re not going to try and make money off of the production
resources. We want to make money off of making it really easy. So, really, what
we want to do is encourage you to, then, go and spin up your own Kubernetes
cluster, say, on Google. And we’ll make a nice little link that says, "Go and
spin up a cluster on GKE." We’ll make that really, really easy, but to make it
super easy, for some number of days, we can just provide you that cluster,
Feature flags – #779
Feature flags are really about decoupling delivery from deployment. It’s the
idea that you make your code, you deploy it, but you haven’t turned it on, so
it’s not delivered yet. And the idea there is that it means you can merge in the
main line, more often, because it’s not affecting anybody. And, also, it really
helps because you can do things like: when I do deliver, I can deliver it for
certain people; just GitLab employees or just the Beta group, and then I can
control that rollout. So then, if there's an error rate spike, well, it’s just
a few a people and I know who they are, and they’re going to complain to me.
It’s no big deal. But I can test things out, get it polished, fix the problems,
before rolling it out. And then, you can also do things like, roll it out to 10 percent
of the people, 50 percent of the people, whatever. It’s all about reducing risk, and
improving quality, and fundamentally about getting things into your mainline
quicker. So, it’s ops-ish, in that sense, but it’s, really, still pretty fully
Artifact management – #2752
Artifact management has become a hot topic lately. We already have a container
registry for Docker image artifacts, and we also have file-based artifacts that
you can pass between jobs, and pass between pipelines, and even pass between
cross project pipelines. And we have ways to download them, and browse them, but
if those artifacts happen to be things like Maven or Ruby or node modules, and
you want to publish them, and then consume them in other pipelines, we don’t
have a formal way to do that.
And you could, obviously, publish to the open source, RubyGems, for example. But
if you want a private Gem, that is only consumed by your team... Maybe that's
not as big for Ruby developers, but Java developers do that all the time. A lot
of Java developers use Artifactory or Sonatype Nexus. In order to complete the
DevOps tool chain, we need to have some first class support for that, either by
bundling in one of these other providers, or by adding layers, and APIs, on top
of our existing artifacts. My personal pet favorite right now is, let’s say we
can just tag our existing artifact, and say, "Oh, this is Maven type of
artifact," and then we expose that via an API and so then you can declare that
in another project, and it would just consume the APIs, and just know how to do
that. But it would also use our built-in authentication so you don’t have to set
up creds and do all this declaration; you can be like, "Oh, I’ve got access to
this project and this project, so I can get the artifacts, and I can consume it
all really easily."
Auto DevOps – #35712
Note: We shipped the first iteration of Auto DevOps in 10.0
So, let’s talk about Auto DevOps. This spans from the near-term to the very
long-term. It’s great that we do a lot of DevOps, and in a very simplistic way,
it’s like, "Oh, but shouldn’t we just make this stuff automatic?" The way I
phrase it is, we should provide the best practices in an easy and default way.
You can set up a GitLab CI YAML, but you have to actively go and do that. But,
really, every project should be running some kind of CI. So, why don’t we just
detect when you’ve pushed up a project; we’ll just build it, and we’ll go and
test it, because we know how to do testing. Today, with Auto Deploy, we already
use Auto Build, with build packs. We will automatically detect, I think, one of
seven different languages, and automatically build your Java app, or Ruby, or
Node... and we use Heroku’s build packs, actually, to do this build. And so we
build that up, and when using Auto Deploy, we’ll go ahead and deploy that. You
still have to, obviously, have a Kubernetes cluster in order to do that, so it’s
not fully automated if you don’t have that. But if you’ve got Kubernetes, hey,
this is a literally one click. You pick from a menu, say, "Oh, I’m on
Kubernetes," and then hit submit, and you’ve got Auto Deploy and Auto Build.
But one of the things we don’t have is Auto CI. And that’s a little annoying,
but it’s one of the things we want to pick up, and actually, hopefully our CTO,
Dmitriy, is going to pick that up in Q3; it's one of his OKRs. Heroku,
themselves, actually extended build packs to do testing, and so that means that
there’s at least five build packs that know how to test these languages. And so,
hey, let’s use that. But even if that doesn’t work, there’s a lot of other
things we can do. Other companies have all this stuff automated, as well. So if
we can’t use Heroku CI, being able to say, "Oh, this is this language; we know
how to test this language," we'll be making that automatic.
Automatic is multiple levels of things. Is it a wizard that configures this
stuff for me? Is it one click checkbox, that says, "Yes, turn on auto CI," or is
it templates that I can easily add into my GitLab CI YAML? I think, in order to
qualify as auto, what we have to do here is that it shouldn’t be templates. It
shouldn’t be blog posts that tell me how to do it. That’s just CI. It should be,
literally, just "I pushed and it worked;" or at most a checkbox or two.
Let’s go further, what other thing could we just automate here? And not automate
strictly for the purposes of automation, but about bringing best practices to
people. So, you have to actively work hard, to turn these things off. If you
don’t want CI, then shut it off, but by default you should have this.
So, this is a really, really long list of things that will take us forever to
get to. The first ones have links, because we’re tracking real issues for this.
Auto Metrics is a great one. If you’re running certain languages, you should
just be able to, really easily, go and just pull the right information out of
there. But whatever, the list is huge.
But the idea is that we can build up this Auto DevOps, even the marketing term,
and start talking about it in that way, and to not just say that GitLab is great
for your DevOps and is a complete DevOps tool chain. But, in fact, we do all
this stuff for you automatically.
There’s a lot to be done to make this fully automated. And what percentage of
projects can we really do? Auto Deploy is a great example that only works for
web apps. If it’s not a web app, we can’t just deploy it. What would it mean? We
deploy it, and it just wouldn’t function. If you made a command line app, what
would deploy even mean? Or if it’s a Maven, or really any kind of module that
you bundled up and released, that’s not the same thing as a deploy. So, maybe we
need an Auto Release. It’s not on this list, but maybe it should be. But within
the web app space, we can do some of this stuff automatically.
So that’s it. Everything you ever wanted to know about DevOps.