A bug when job
needs a manual job
In 13.12 we fixed a bug that might affect the existing behavior of your pipeline. We explain why we had to fix the bug, the possible impact of this change on your pipeline, and the proposed workaround if you would like to revert this behavior.
Background on a two-job pipeline
In GitLab CI/CD you can easily configure a job to require manual intervention before it runs. The job gets added to the pipeline, but doesn't run until you click the play button on it.
Let's look at a two-job pipeline:
stages: - stage1 - stage2 job1: stage: stage1 script: - echo "this is an automatic job" manual_job: stage: stage2 script: - echo "This is a manual job which doesn't start automatically, and the pipeline can complete without it starting." when: manual # This setting turns a job into a manual one
This is how it looks when we look at the pipeline graph:
Notice that the manual job gets skipped, and the pipeline completes successfully even though the manual job did not get triggered. This happens because manual jobs are considered optional, and do not need to run.
Internally, manual jobs have
allow_failure set to true by default, which means that these skipped manual jobs do not cause a pipeline failure. The YAML code below demonstrates how to write the manual job, which results in the same behavior. The job doesn't automatically start, is skipped, and the pipeline passes.
manual_job: stage: stage2 script: - echo "This is a manual job which doesn't start automatically, and the pipeline can complete without it starting." when: manual allow_failure: true # this line is redundant since manual job has this setting by default
You can set
allow_failure to true for any job, including both manual and automatic jobs, and then the pipeline does not care if the job runs successfully or not.
How to expand the configuration with
Last year we introduced the
needs keyword which lets you create a Directed Acyclic Graphs (DAG) to speed up your pipeline. The
needs keyword creates a dependency between two jobs regardless of their stage.
Let's look at this example:
stages: - stage1 .... - stage10 job1: # this is the first job that runs in the pipeline stage: stage1 script: - echo "exit 0" ..... job10: needs: # Defined a "needs" relationship with job1 - job1 stage: stage10 script: - echo "This job runs as soon as job1 completes, even though this job is in stage10."
needs keyword creates a dependency between the two jobs, so
job10 runs as soon as
job1 finishes running successfully, regardless of the stage ordering.
So what happens if a job
needs a manual job, that doesn't start running automatically?
Let's look at the following example:
stages: - build - test - deploy build: stage: build script: exit 0 test: stage: test when: manual script: exit 0 deploy: stage: deploy script: echo "when should this job run?" needs: - test
Before 13.12, this type of configuration would cause the pipeline to get stuck. The
deploy job can only start when the
test job completes, but the
test job does not start automatically. The rest of the pipeline stops and waits for someone to run the manual
This behavior is even worse with larger pipelines:
The example above shows there is a needs relationship between
post test job and the
test job (which is a manual job) as you can see the pipeline is stuck in a running state and any subsequent jobs will not run.
This was not the behavior most users expected, so we improved it in 13.12. Now, if there is a
needs relationship pointing to a manual job, the pipeline doesn't stop by default anymore. The manual job is considered optional by default in all cases now. Any jobs that have a
needs relationship to manual jobs are now also considered optional and skipped if the manual job isn't triggered. If you start the manual job, the jobs that need it can start after it completes.
Note that if you start the manual job before a later job that has it in a
needs configuration, the later job will still wait for the manual job to finishes running.
What if I don't want this new behavior?
One of the reasons we selected this solution is that you can quickly revert this change. If you made use of this inadvertent behavior and configured your pipelines to use it to block on manual jobs, it's easy to return to that previous behavior. All you have to do is override the default
allow_failure in the manual job with
allow_failure: false. This way the manual job is no longer optional, and the pipeline status will be marked as blocked and wait for you to run the job manually.
stages: - build - test - deploy build: stage: build script: exit 0 test: stage: test when: manual allow_failure: false # Set to false to return to the previous behavior. script: exit 0 deploy: stage: deploy script: exit 0 needs: - test
Share any thoughts, comments, or questions, by opening an issue in GitLab and mentioning me (
“Text to tweet” – Dov Hershkovitch
Click to tweet