This is a proposal to extend GitLab CI with parallel builds, pushing successful builds to external repo’s, branch specific builds and support for build metrics.
GitLab CI is quickly becoming more popular and people appreciate the good integration with GitLab and the clean UI.
This popularity also brings requests for the following functionality that make a lot of sense:
The parallel builds feature will allow tests to complete a lot faster. Suppose you have the following build:
With a total build time of 20 minutes. With a parallel build and two runners you can have the following:
For a total build time of 12 minutes. The parallel builds are almost two times as fast.
The days when we just run a single test type on our software are long gone. We look forward to a future where in the merge request you see the following:
In order to achieve the requirements the following things need to change in the GitLab CI coordinator:
The script types mentioned in the last item have the following characteristics:
Suppose a project has 7 scripts:
If there is a new commit on the project the CI coordinator will create one Commit object and two Build objects, one for each of the test scripts. When a runner requests one of these it will receive the concatenated contents of the two setup scripts and one of the test scripts. The two runners report back their progress by submitting their logs to the CI coordinator just like today. Suppose that both Builds complete successfully. An observer on the CI coordinator will now create two new Build objects for the success scripts. These can again run on different runners.
To ensure a for example a Jekyll blog is quickly deployed people should put both the generation and the deployment into the success task. Having the generation in a test and the deployment in a success script is not an option because the scripts could run on different runners. Repeating the generation in a test and the success script slows down the complete process since another runner has to do the setup and the test before the success task can run. If the generation fails in the success task this runner will exit with a failure before the success task is run as long as the generation and deployment are on different lines of the script.
Maybe you noticed that the example merge request content talked about ‘Older Ruby’ and ‘Another DB’. One consequence of this proposal is that we will not support a build matrix where all combinations of options are tested. So you can’t easily test your project with all combinations of 3 ruby versions, 2 databases, 4 gemfiles and 3 environments. We recommend to choose a default configuration (ruby 2.0, PostgeSQL, Gemfile, production) for which you run the specific test scripts (rspec, cucumber, etc.). The other options have a script run all tests (for example rake:test) with just that option changed. This means that you miss failures that are a result of two options (a different ruby and a different database). In our experience most failures can already be caught by testing one option at a time. As a workaround you can of course manually create scripts for combinations of options. The advantage of not testing combinations is that you need a lot less testing, 12 complete builds instead of 72 in our example. Another advantage is that executing the specific test scripts in parallel allow you to quickly identify where an error occurs.
Apart from giving each script a type they can also have a branch restriction. This means the script only runs if the current branch matches on of the branches listed in the branch restiction. For example you can only deliver a failure notification if the master or production branch breaks. The current branch is already passed from GitLab to GitLab CI coordinator when a new commit is pushed.
Percentage results as output are something that is useful for only part of the tests. For tests like Rspec and Cucumber a pass/fail result is enough. But for code quality tools that measure things such as test coverage, code complexity and code style a percentage is appropriate. For example for a code coverage tool like simplecov a 81.45% code coverage is much more useful than a pass/fail result. Adding percentage results means that the runner has to determine the percentage somehow. We propose the use the CI_RESULT environmental variable for that. So when you run a code quality tool you should use a regular expression to parse the results and set CI_RESULT to a decimal value (for example 0.2345 to represent 23.45%). Some tools do not output a percentage but an absolute number, for example Rails perftest outputs the number of milliseconds something took. In this case you have to normalize the output. One way to do this would be to select a value that is perfect (25ms for a page load) and diving that by the output. Suppose the page loads in 67ms, the normalized result is 25/67 = 0.3731 = 37.31%. The advantage of normalizing in this way is that high percentages are better a slight disadvantage might be that you can get values over 100%. GitLab CI coordinator can generate graphs of how the percentages of the master build for each individual script change over time. Even when CI_REULT is set the runner always also submits the pass/fail result based on the exit code to the coordinator. Obviously old runners will not submit this value to the coordinator, in that case the coordinator falls back on the pass/fail result.
Even with the help of the rest of the community implementing the above will take multiple versions of GitLab CI. We could use the following implementation steps:
We think the above is a feasible way to integrate a lot of useful features in GitLab CI. We look forward to hear your questions, comments and suggestions.
Update on 2014-03-26: As commented below we would like to also include the ability to assign a runner to a specific script. This would enable you to perform a performance test on a specific machine or to perform a build on multiple operating systems.Install GitLab on your own server in 2 minutes