Data Triage Guide

1. You are here:
2. Business Technology
3. Data Team
4. Data Team - How We Work
5. Data Triage Guide

Maintained by:

G

On this page

• Data Triage
  • Enterprise Data Program Triage
  • Enterprise Data Triage Groups
  • Enterprise Data GitLab Projects
  • Enterprise Data Triager
    • Data Platform weekly rotation schedule
  • Enterprise Data Program Triage Instructions
    • Triage responsibilites
  • How to Complete Data Triage
  • Incident
    • Severity
    • Follow up
  • Triage Bot
  • End of day wrap-up
  • End of week wrap-up
  • GitLab.com DB structure changes
    • Graphical representation of the process
• Triage common issues
  • GitLab Postgres Database not accessible
  • Automated service ping issue
  • Zuora Stitch Integration single or set of table-level reset
    • Step 1:- Rename existing table with the date suffix to identity the backup, recommended format YYYYMMDD
    • Step 2:- Pause the regular integration.
    • Step 3:- Create a new integration Zuora-Subscription in Stitch.
    • Step 4:- Run the newly created integration..
    • Step 5:- Check for the records.
    • Step 6:- Create the table in the main schema.
    • Step 7:- Make records count check to ensure we don't have fewer records in the new table.
    • Step 8:- Drop the new schema
    • Step 9:- Delete temp Zuora-Subscription integration and enable regular integration
    • Step 10:- Run regular integration and validate
  • Source freshness errors
  • Airflow Task failure
  • Sheetload - Column '#REF!' is not recognised
  • Model version_usage_data_unpacked stale
  • Zuora Revenue Source and Target Column Mismatch
• Triage FAQ
• Useful regex
  • Match lines where these terms do not exist

Data Triage

Enterprise Data Program Triage

GitLab has a robust and vibrant Data Program which includes a Central Data Team and many Functional Analytics Teams. GitLab total team members are growing as well and we need to uplevel our triaging process to keep up with GitLab's growth.

Steps to uplevel triaging process:

1. Incorporate Functional Analytics Teams into the #data channel triaging process.
2. Create Slack Aliases so that GitLab Team members can ping a group of folks to get help with their questions in #data.
3. Create a Triage Schedule for the Data Program to follow on the Data Program Calendar. Ask for a commitment of one team member, per Business Team, per day to help triage.

Enterprise Data Triage Groups

Enterprise Data GitLab Projects

Enterprise Data Triager

Data Platform weekly rotation schedule

The Data Platform Team will run an experiment in FY23-Q4 to have a weekly rotation schedule. This means that a Data Platform Team member is performing triage responsibilities from Monday - Friday. With running a weekly schedule Team members could focus more and better on outstanding issues and incidents and prevent doing handovers each and every day.

Before the start of the quarter the Data Platform Team alligns on the weekly rotation schedule via an issue.

• FY23-Q4: https://gitlab.com/gitlab-data/analytics/-/issues/14815

Because of public holidays or unforeseen circumstances (Family and Friends day excluded, on these days we do not perform triage), the schedule can change throughout the quarter where there is still a possibility to handover a single day amongst Data Platform Team members. These changes are arranged between team members and are reflected in the Data Team calendar.

At the end of FY23-Q4 we will evaluate the experiment and decide how we will do triage in the Data Platform Team moving forward. As we try to make this fact based and data driven we will look at the following KPIs:

• Average completion time of issues and incidents
• Close ratio of issues and incidents

Second we will also run a pulse check amongst Data Platform Team members to hear the general impression of the new triage process. The combination of both will determine how we move forward.

Enterprise Data Program Triage Instructions

1. GitLab Team members can ping one of the groups above for any questions or help. When not clear, ping a group based on best guess. If a different group was needed, the request or question can always be re-routed.
2. Triager should review each slack message request in the #data that has the triager's assigned Slack alias pinged.
3. Triager should reply to slack threads by pointing GitLab team member to the appropriate handbook page, visualization, or to other GitLab team members who may know more about the topic.
4. Triager should direct GitLab team member to the channel description, which has links to the various Data Program Projects, if the request requires more than 5 minutes of investigative effort from a Data Program team member.

A team member who is off, on vacation, or working on a high priority project is responsible for finding coverage and communicating to the team who is taking over their coverage; this should be updated on the Data Program's Google Calendar. To avoid putting the Monday workload on the same shoulders every week again, the Data Platform Team will rotate/exchange every now and then triage days in good collaboration on an ad-hoc basis.

Having dedicated triagers on the team helps address the bystander affect. The schedule shares clear daily ownership information but is not an on-call position. Through clear ownership, we create room for everyone else on the team to spend most of the day around deep work. The triager is encouraged to plan their day for the kind of work that can be accomplished successfully with this additional demand on time.

Triage responsibilites

Data triagers are the first responders to requests and problems for the Data Program.

• The Data Analyst triager is primarily responsible for responding to GitLab team member requests whether via issue, which posts to #data-triage, or directly via slack in #data.
• The Functional Analyst triager is primarily responsible for responding to GitLab team member requests via slack in #data.
• The Data Platform triager is responsible for resolving problems with our data platform, which will either be in the #data-pipelines and #data-prom-alerts slack channels and on the DE - Triage Errors board. Issues created from these errors should use the DE Triage Errors issue template.
  • During the assigned triage week the Data Platform Team member will focused on (in priority order):
    • incoming incidents
    • open incidents
    • new issues
    • open issues on the Data Platform - Triage Errors board.
      • If an open incident or issue is already assigned it is still the triager responsibility to either take that issue or ensure progress is made.
      • If there is no work to be performed on incidents or issues on the board the triager will work on their regular work assignments.
  • Involvement from Data Platform Team members who do not carry triage responsibilities that week is likely still needed in somes cases like:
    • A standing issue or incident could not be solved by the triager and triager need help from other Data Platform Team members.
    • Monitoring #data-prom-alerts:
      • The #data-prom-alerts slack channel is used for the most urgent breaking events, which requires immediate action. It is the responsibility of all Data Platform Team members to ensure action is taken in time after office hours of the triager.
    • Assistance from the Data Platform Team is needed by other GitLab Team members and this is outside of the office hours of the triager.
• The Analytics Engineer triager is primarily responsible for resolving dbt-test and dbt-run errors. Issues created from these errors should use the AE Triage Errors issue template.
• For more information on responsibilities of a triager watch the Data Engineer triage training session video.

Data Analysts
- Respond to #data questions and assign to right Triage group
- Respond to new unassigned issues in our project and assign to the right team

Data Engineers:
- Respond to data infrastructure incidents
- Respond to broken data pipelines
- Respond to data freshness issues
- Respond to broken dags (except for dbt issues, see next)

Analytics Engineers:
- Respond to errors in dbt models
- Respond to dbt test failures

How to Complete Data Triage

For issue triaging, the triager should add the appropriate labels to the issue to put the issue into the respective project's workflow. It is not the responsibility of the triager to validate the issue and determine any root causes or solutions. Rather, for the Data Team project for example, the triager should add the appropriate scoped TEAM label (Ex.Team:GTM) and the scoped Workflow::triage label. From there, those issues are then put into the respective team's workflow.

Note: The Triager:

• is not expected to know the answer to all the questions
• should cc (mention) other team members who are subject matter experts (SME) or functional DRIs for their feedback
• should document any issues stumbled upon and help to disseminate knowledge amongst all team members

The Central Data Team triager will create an issue in the Data Team project. Task and duties are stated in the issue template.

Read the FAQ and common issues.

Incident

The Data Team follows the incident definition from Engineering: Incidents are anomalous conditions that result in—or may lead to—service degradation or outages. These events require human intervention to avert disruptions or restore service to operational status.

Service degradation or outages in data can be seen as:

• Data is not available
• Data is not correct
• Data is outdated
• Data leakage

This means the following events (not extensive) are likely to be incidents:

• DBT model failed (and downstream models are skipped)
• DBT test failure
• DBT source freshness failure
• Infrastructural incident (Snowflake not accessible, Sisense not accessible)
• Data source not available
• Data pipeline not running

Severity

Depending on the nature and impact of the incident a severity needs to be determined. Currently we don't have a decision matrix in place. To determine the severity, take the following aspects into consideration:

• In case of a data source or data pipeline incident, check the Data Source Tier.
• Number of users impacted
• Impacting Trusted Data models

Follow up

• Incidents are always given immediate attention.
• Every incident has a DRI assigned. This is not necessarily the triager/creator of the incident. Due to the nature of asynchronous working at GitLab, the triager/creator is the DRI until another GitLab Team Member is actively contacted/involved.
  • The codeownerfile is the right future* source to find the right DRI for assigning the incident. * Currently the code ownership is not well defined. As part of FY23-Q1 we are planning to have a more strict ownership.
• Every raised incident will be communicated in the #data Slack channel, followed by a short description, ETA and link to the incident. The right GitLab Team Members are tagged.
  • A regular (depending on the severity) update is posted in Slack. Sometimes there isn't a new status, don't hesitate to communicate this as well.
  • Timelines should be documented in the timeline section under the incident Subject/Header for use in retrospectives and other investigations.
  • When the incident is solved, an update is posted in Slack

Triage Bot

Parts of triage are assisted by the GitLab Triage Bot, which is setup in the Analytics project. The bot runs every hour and takes actions based on a set of rules defined in a policies file. The GitLab Triage README contains all documentation for the formatting and definition of rules.

Changes to the triage bot policy file should be tested in the MR by running the "dry-run:triage" CI job and inspecting the log output. This CI job is a dry-run, meaning it will not actually take any actions in the project but will print out what would happen if the policy was actually executed.

End of day wrap-up

In order to get better and be more efficient in daily triage, we wrap-up the work by the end of the day. The following information is provided by the Data Analyst each day:

• Time Tracking: Listing down the time spent that day on triage and which activities are performed. The general idea is to gain an understanding on where the workload is, in order to optimize those activities. Please list down the work so it is useful for analysis purposes. I.e:
  • 2 hours solving data ingestion issue, for source x
  • 1 hours solving data transformation issue
  • 1 hour on solving a broken Sisense Dashboard
  • 1 hour triaging and re-routing (new) issues
  • 0.5 hour on answering Slack questions
• Groundhog Issues. Issues that occur on a regular basis are annoying and costing us unnecessary time. List down any issue that is popping up time after time. This is in order to find any spots in the current landscape to address.

A triage roundup will take place at the end of every milestone by the data leadership team to consolidate the milestones triage efforts. Please bear in mind the purpose of the information provided, to make it useful and improve Triage.

End of week wrap-up

The Data Platform Team follows a weekly rotation schedule which means that by the end of the triage week the triager will hand over the triage responsibilities.

• Even though we still use the triage issue on a daily basis, the triager will only write up the week a-sync in the central data team triage issue on Friday end of day.
• The triager will report/verbalize in the Weekly Data Platform Team meeting any notable things happened on triage on Tuesday in the next week.

Although running a weekly rotation, we expect the triager to post an EOD announcement in the applicable Slack channels.

GitLab.com DB structure changes

1 of the most important data source, that regularly changes, is the GitLab.com database. In order not to break the daily operation, changes to the database needs to be tracked and checked. Any change to the GitLab.com database, is made to the db/structure.sql file. The Data Team gets notified, by applying labels to the MR, if a change to the db/structure.sql is made, via the Danger Bot.

A label Data Warehouse::Impact Check is added by the Danger Bot as call to action for the data team.

• On triage, the Triager will check for MRs with label Data Warehouse::Impact Check.

The following actions are performed by Data Team Triager:

• Every merge request (MR) will be judged
  • If MR contains the label group::product intelligence along with Data Warehouse::Impact Check, there are a couple of checks that need to do:
    • Because a new metric is added or the existing one is altered, the Data team should ensure the change will not break the Service ping extraction process
    • Check new metric SQL statement from the original MR (a typical example is gitlab-org/gitlab/merge_requests/75504) and execute it on Snowflake - usually, it is just a SELECT SQL statement
  • If the changes to the SQL file are not causing a break in the operation, the label will be changed to Data Warehouse::Not Impacted.
  • If the changes to the SQL file causing a break in the operation:
    • The Label will be changed to Data Warehouse::Impacted
    • A new issue is opened in the GitLab Data Team project, assigned to the correct DRI and linked to the original MR.
    • Impact will be determined in the issue.
    • Any MRs will be created to overcome loading issues, downstream dbt processing and Sisense usage.
    • According to the Merge of the GitLab.com MR, merge will be planned.
  • If the MR does not contains the label group::product intelligence and it concerns changes to SQL structure:
    • Check if it will break the operation / data pipeline, following the Determination matrix below.
  • If any MR will cause a break in the operation, the label will be changed to Data Warehouse::Not Impacted.
  • If any MR will cause a break in the operation:
    • The Label will be changed to Data Warehouse::Impacted
    • A new issue is opened in the GitLab Data Team project, assigned to the correct DRI and linked to the original MR.
    • Impact will be determined in the issue.
    • Any MRs will be created to overcome loading issues, downstream dbt processing and Sisense usage.
    • According to the Merge of the GitLab.com MR, merge will be planned.
    • All stakeholders will be informed.

Graphical representation of the process

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  the database system is starting up\n
b'FATAL:  the database system is starting up\n'

ALTER TABLE "RAW"."ZUORA_STITCH"."SUBSCRIPTION" RENAME TO "RAW"."ZUORA_STITCH"."SUBSCRIPTION_20210903";

CREATE TABLE "RAW"."ZUORA_STITCH"."SUBSCRIPTION" CLONE  "RAW"."ZUORASUBSCRIPTION"."SUBSCRIPTION"; **Note:** Check for the primary key present in the table post clone or not if not check for the primary key in the [link](https://www.stitchdata.com/docs/integrations/saas/zuora#subscription) and add the constraints on those columns. 

select count(*) from "RAW"."ZUORA_STITCH"."SUBSCRIPTION_20210903" where deleted = 'FALSE';
select count(*) from "RAW"."ZUORA_STITCH"."SUBSCRIPTION" ;

DROP SCHEMA "RAW"."ZUORASUBSCRIPTION"  CASCADE ;

select id, count(*) from "RAW"."ZUORA_STITCH"."SUBSCRIPTION"
group by id
having count(*) > 1 **Note** Refer to the [MR (internal link)](https://gitlab.com/gitlab-data/analytics/-/issues/10065#note_668365681) for more information.

[2022-01-26 11:56:32,233] INFO - b'\x1b[33mDatabase Error in model version_usage_data_unpacked (models/legacy/version/xf/version_usage_data_unpacked.sql)\x1b[0m\n'
[2022-01-26 11:56:32,233] INFO - b' 000904 (42000): SQL compilation error: error line 241 at position 12\n'
[2022-01-26 11:56:32,233] INFO - b" invalid identifier '{metrics_name}'\n"
[2022-01-26 11:56:32,233] INFO - b' compiled SQL at target/compiled/gitlab_snowflake/models/legacy/version/xf/version_usage_data_unpacked.sql\n'
[2022-01-26 11:56:32,234] INFO - b'\n'

[2022-03-21, 13:26:48 UTC] INFO - sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 100080 (22000): 01a31326-0403-d02c-0000-289d37f10d4e: Number of columns in file (102) does not match that of the corresponding table (99), use file format option error_on_column_count_mismatch=false to ignore this error
[2022-03-21, 13:26:48 UTC] INFO -   File 'RAW_DB/staging/BI3_RC_POB/BI3_RC_POB_12.csv', line 3, character 1
[2022-03-21, 13:26:48 UTC] INFO -   Row 1 starts at line 2, column "BI3_RC_POB"[102]
[2022-03-21, 13:26:48 UTC] INFO -   If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client.

gsutil cp  gs://zuora_revpro_gitlab/RAW_DB/staging/BI3_RC_POB/BI3_RC_POB_1.csv .

head -1 BI3_RC_POB_1.csv 

USE ROLE LOADER;
USE DATABASE RAW;
USE SCHEMA ZUORA_REVENUE;
select get_ddl('table','BI3_RC_POB');

gsutil cp gs://zuora_revpro_gitlab/RAW_DB/processed/22-03-2022/BI3_RC_POB/BI3_RC_POB_22-03-2022.log  gs://zuora_revpro_gitlab/RAW_DB/staging/BI3_RC_POB/