A bug when job needs a manual job

In 13.12 we fixed a bug that might affect the existing behavior of your pipeline. We explain why we had to fix the bug, the possible impact of this change on your pipeline, and the proposed workaround if you would like to revert this behavior.

Background on a two-job pipeline

In GitLab CI/CD you can easily configure a job to require manual intervention before it runs. The job gets added to the pipeline, but doesn't run until you click the play button on it.

Let's look at a two-job pipeline:

stages:
  - stage1
  - stage2

job1:
  stage: stage1
  script:
    - echo "this is an automatic job"

manual_job:
  stage: stage2
  script:
    - echo "This is a manual job which doesn't start automatically, and the pipeline can complete without it starting."
  when: manual # This setting turns a job into a manual one

This is how it looks when we look at the pipeline graph:

image2

Notice that the manual job gets skipped, and the pipeline completes successfully even though the manual job did not get triggered. This happens because manual jobs are considered optional, and do not need to run.

Internally, manual jobs have allow_failure set to true by default, which means that these skipped manual jobs do not cause a pipeline failure. The YAML code below demonstrates how to write the manual job, which results in the same behavior. The job doesn't automatically start, is skipped, and the pipeline passes.

manual_job:
  stage: stage2
  script:
    - echo "This is a manual job which doesn't start automatically, and the pipeline can complete without it starting."
  when: manual
  allow_failure: true # this line is redundant since manual job has this setting by default

You can set allow_failure to true for any job, including both manual and automatic jobs, and then the pipeline does not care if the job runs successfully or not.

How to expand the configuration with needs (DAG)

Last year we introduced the needs keyword which lets you create a Directed Acyclic Graphs (DAG) to speed up your pipeline. The needs keyword creates a dependency between two jobs regardless of their stage.

Let's look at this example:

stages:
  - stage1
  ....
  - stage10

job1: # this is the first job that runs in the pipeline
  stage: stage1
  script:
    - echo "exit 0"
.....

job10:
  needs:  # Defined a "needs" relationship with job1
    - job1
  stage: stage10
  script:
    - echo "This job runs as soon as job1 completes, even though this job is in stage10."

The needs keyword creates a dependency between the two jobs, so job10 runs as soon as job1 finishes running successfully, regardless of the stage ordering.

So what happens if a job needs a manual job, that doesn't start running automatically?

Let's look at the following example:

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: exit 0

test:
  stage: test
  when: manual
  script: exit 0

deploy:
  stage: deploy
  script: echo "when should this job run?"
  needs:
    - test

Before 13.12, this type of configuration would cause the pipeline to get stuck. The deploy job can only start when the test job completes, but the test job does not start automatically. The rest of the pipeline stops and waits for someone to run the manual test job.

image3

This behavior is even worse with larger pipelines:

image4

The example above shows there is a needs relationship between post test job and the test job (which is a manual job) as you can see the pipeline is stuck in a running state and any subsequent jobs will not run.

This was not the behavior most users expected, so we improved it in 13.12. Now, if there is a needs relationship pointing to a manual job, the pipeline doesn't stop by default anymore. The manual job is considered optional by default in all cases now. Any jobs that have a needs relationship to manual jobs are now also considered optional and skipped if the manual job isn't triggered. If you start the manual job, the jobs that need it can start after it completes.

Note that if you start the manual job before a later job that has it in a needs configuration, the later job will still wait for the manual job to finishes running.

What if I don't want this new behavior?

One of the reasons we selected this solution is that you can quickly revert this change. If you made use of this inadvertent behavior and configured your pipelines to use it to block on manual jobs, it's easy to return to that previous behavior. All you have to do is override the default allow_failure in the manual job with allow_failure: false. This way the manual job is no longer optional, and the pipeline status will be marked as blocked and wait for you to run the job manually.

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: exit 0

test:
  stage: test
  when: manual
  allow_failure: false  # Set to false to return to the previous behavior.
  script: exit 0

deploy:
  stage: deploy
  script: exit 0
  needs:
    - test

Share any thoughts, comments, or questions, by opening an issue in GitLab and mentioning me (@dhershkovitch).

Try all GitLab features - free for 30 days

GitLab is more than just source code management or CI/CD. It is a full software development lifecycle & DevOps tool in a single application.

Try GitLab Free
Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license

Try GitLab risk-free for 30 days.

No credit card required. Have questions? Contact us.

Gitlab x icon svg