Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Geo scheduled pipelines

  1. You are here:
  2. Engineering
  3. Development Department
  4. Enablement Sub-department
  5. Enablement:Systems Stage
  6. Geo and Disaster Recovery
  7. Geo scheduled pipelines

On this page

Using a combination of GET and gitlab-qa, a series of scheduled pipelines are periodically run to check if there are any breaking changes in Geo. It is necessary to check the Alerts to check for errors, and handle them as appropriate.

Pipelines

Title Description Frequency
1k QA Runs Geo QA steps against a 1k Reference Architecture Inactive
3k QA Runs Geo QA steps against a 3k Reference Architecture Inactive
1k Failover & Recovery Tests failover on a 1k Reference Architecture Inactive
1k Rebuild Destroys and recreates the 1k Reference Architecture Daily at 4AM UTC
3k Rebuild Destroys and recreates the 3k Reference Architecture Mondays at 4AM UTC
3k Update Upgrades Gitlab on the 3k Reference Architecture Tuesday through Friday at 4AM UTC

Handling

  1. Check the alert list for triggered, unassigned alerts
  2. Acknowledge any unhandled alert, and assign it to yourself
  3. Check the failed pipeline to determine a cause of the failure
    1. All of the instances are built in the gitlab-qa-geo-ci GCP group. Geo engineers should have access to the group, and can log onto the individual VMs to access the logs as required.
  4. If there is no issue, mark the alert as resolved
  5. If there is an issue, create an incident to track the resolution
  6. If multiple alerts are being triggered for the same reason, only create one incident
  7. If it is a simple fix, fix and close the incident
  8. If it is not a simple fix
    1. Open a follow up issues to handle the fix
    2. Quarantine the failing test
    3. Close the incident
Open in Web IDE View source