Primary Project dbt docs Epics OKRs
We Data
The Data Team is a part of the Finance organization within GitLab, but we serve the entire company. We do this by maintaining a data warehouse where information from all business systems are stored and managed for analysis.
Our charter and goals are as follows:
The Data Team at GitLab is working to establish a world-class analytics function by utilizing the tools of DevOps in combination with the core values of GitLab. We believe that data teams have much to learn from DevOps. We will work to model good software development best practices and integrate them into our data management and analytics.
A typical data team has members who fall along a spectrum of skills and focus. For now, the analytics function at GitLab has Data Engineers and Data Analysts; eventually the team will include Data Scientists. Analysts are divided into being part of the Core Data Function and specializing in different functions in the company.
Data Engineers on our team are essentially software engineers who have a particular focus on data movement and orchestration. The transition to DevOps is typically easier for them because much of their work is done using the command line and scripting languages such as Bash and Python. One challenge in particular are data pipelines. Most pipelines are not well tested, data movement is not typically idempotent, and auditability of history is challenging.
Data Analysts are further from DevOps practices than Data Engineers. Most analysts use SQL for their analytics and queries, with Python or R a close second. In the past, data queries and transformations may have been done by custom tooling or software written by other companies. These tools and approaches share similar traits in that they're likely not version controlled, there are probably few tests around them, and they are difficult to maintain at scale.
Data Scientists are probably furthest from integrating DevOps practices into their work. Much of their work is done in tools like Jupyter Notebooks or R Studio. Those who do machine learning create models that are not typically version controlled. Data management and accessibility is also a concern as well.
We will work closely with the analytics community to find solutions to these challenges. Some of the solutions may be cultural in nature, and we aim to be a model for other organizations of how a world-class Data and Analytics team can utilize the best of DevOps for all Data Operations.
Some of our beliefs are:
Analysis usually begins with a question. A stakeholder will ask a question of the data team by creating an issue in the Data Team project using the appropriate template. The analyst assigned to the project may schedule a discussion with the stakeholder(s) to further understand the needs of the analysis. This meeting will allow for analysts to understand the overall goals of the analysis, not just the singular question being asked, and should be recorded. Analysts looking for some place to start the discussion can start by asking:
An analyst will then update the issue to reflect their understanding of the project at hand. This may mean turning an existing issue into a meta issue or an epic. Stakeholders are encouraged to engage on the appropriate issues. The issue then becomes the SSOT for the status of the project, indicating the milestone to which it has been assigned and the analyst working on it, among other things. Barring any confidentiality concerns, the issue is also where the final project will be delivered. On delivery, the data team manager will be cc'ed where s/he will provide feedback and/or request changes. When satisfied, s/he will close the issue. If the stakeholder would like to request a change after the issue has been closed, s/he should create a new issue and link to the closed issue.
The Data Team can be found in the #data channel on slack.
The data team's priorities come from our OKRs. We do our best to service as many of the requests from the organization as possible. You know that work has started on a request when it has been assigned to a milestone. Please communicate in the issue about any pressing priorities or timelines that may affect the data team's prioritization decisions. Please do not DM a member of the data team asking for an update on your request. Please keep the communication in the issue.
The data team currently works in two-week intervals, called milestones. Milestones start on Tuesdays and end on Mondays. This discourages last-minute merging on Fridays and allows the team to have milestone planning meetings at the top of the milestone.
Milestones may be three weeks long if they cover a major holiday or if the majority of the team is on vacation or at Contribute. As work is assigned to a person and a milestone, it gets a weight assigned to it.
Data Team OKRs are derived from the higher level BizOps/Finance OKRs as well as the needs of the team. At the beginning of a FQ, the team will outline all actions that are required to succeed with our KRs and in helping other teams measure the success of their KRs. The best way to do that is via a team brain dump session in which everyone lays out all the steps they anticipate for each of the relevant actions. This is a great time for the team to raise any blockers or concerns they foresee. These should be recorded for future reference.
These OKRs drive ~60% of the work that the core data team does. The remaining time is divided between urgent issues that come up and ad hoc/exploratory analyses. Specialty data analysts (who have the title "Data Analyst, Specialty") should have a similar break down of planned work to responsive work, but their priorities are set by their specialty manager.
Given the priorities of the team, work is then pulled into a milestone and then assigned appropriately. (Work is not assigned and then managed into a milestone.) Given the power of the Ivy Lee method, this allows the team to collectively work on priorities as opposed to creating a backlog for any given person. As a tradeoff, this also means that every time a core analyst is introduced to a new data source their velocity may temporarily decrease as they come up to speed; the overall benefit to the organization that any analyst can pick up any issue will compensate for this, though. Learn how the product managers plan groom issues.
Milestone planning should take into consideration:
Any member of the team currently participating in the milestone can wear the milestone planning hat for two milestones, then pass it along to the next teammate.
The timeline for milestone planning is as follows:
The short-term goal of this process is to improve our ability to plan and estimate work through better understanding of our velocity. In order to successfully evaluate how we're performing against the plan, any issues not raised at the T+7 mark should not be moved until the next milestone begins.
There are three general types of issues:
Not all issues will fall into one of these buckets but 85% should.
Some issues may need a discovery period to understand requirements, gather feedback, or explore the work that needs to be done. Discovery issues are usually 2 points.
Introducing a new data source requires a heavy lift of understanding that new data source, mapping field names to logic, documenting those, and understanding what issues are being delivered. Usually introducing a new data source is coupled with replicating an existing dashboard from the other data source. This helps verify that numbers are accurate and the original data source and the data team's analysis are using the same definitions.
This umbrella term helps capture:
It is the responsibility of the assignee to be clear on what the scope of their issue is. A well-defined issue has a clearly outlined problem statement. Complex or new issues may also include an outline (not all encompassing list) of what steps need to be taken. If an issue is not well-scoped as its assigned, it is the responsibility of the assignee to understand how to scope that issue properly and approach the appropriate team members for guidance early in the milestone.
| Weight | Description |
|---|---|
| Null | Meta and Discussions that don't result in an MR |
| 0 | Should not be used. |
| 1 | The simplest possible change including documentation changes. We are confident there will be no side effects. |
| 2 | A simple change (minimal code changes), where we understand all of the requirements. |
| 3 | A simple change, but the code footprint is bigger (e.g. lots of different files, or tests effected). The requirements are clear. |
| 5 | A more complex change that will impact multiple areas of the codebase, there may also be some refactoring involved. Requirements are understood but you feel there are likely to be some gaps along the way. |
| 8 | A complex change, that will involve much of the codebase or will require lots of input from others to determine the requirements. |
| 13 | A significant change that may have dependencies (other teams or third-parties) and we likely still don't understand all of the requirements. It's unlikely we would commit to this in a milestone, and the preference would be to further clarify requirements and/or break into smaller Issues. |
Think of each of these groups of labels as ways of bucketing the work done. All issues should get the following classes of labels assigned to them:
Optional labels that are useful to communicate state or other priority
| Priority | Description | Probability of shipping in milestone |
|---|---|---|
| P1 | Urgent: top priority for achieving in the given milestone. These issues are the most important goals for a milestone and should be worked on first; some may be time-critical or unblock dependencies. | ~100% |
| P2 | High: important issues that have significant positive impact to the business or technical debt. Important, but not time-critical or blocking others. | ~75% |
| P3 | Normal: incremental improvements to existing features. These are important iterations, but deemed non-critical. | ~50% |
| P4 | Low: stretch issues that are acceptable to postpone into a future milestone. | ~25% |
Members of the data team use Geekbot for our daily standups. These are posted in #data-daily. When Geekbot asks, "What are you planning on working on today? Any blockers?" try answering with specific details, so that teammates can proactively unblock you. Instead of "working on Salesforce stuff", consider "Adding Opportunity Owners for the sfdc_opportunity_xf model`." There is no pressure to respond to Geekbot as soon as it messages you. Give responses to Geekbot that truly communicate to your team what you're working on that day, so that your team can help you understand if some priority has shifted.
Ideally, your workflow should be as follows:
WIP: label. cc @user in a comment.WIP: label, mark the branch for deletion, mark squash commits, and assign to the project's maintainer. Ensure that the attached issue is appropriately labeled and pointed.Other tips:
We use GitLab to operate and manage the analytics function. Everything starts with an issue. Changes are implemented via merge requests, including changes to our pipelines, extraction, loading, transformations, and parts of our analytics.
| Stage | Tool |
|---|---|
| Extraction | Stitch and Custom |
| Loading | Stitch and Custom |
| Orchestration | Airflow and GitLab CI |
| Storage | Cloud SQL (PostgreSQL) and Snowflake |
| Transformations | dbt and Python scripts |
| Analysis | Periscope Data |
We currently use Stitch for most of our data sources.
| Data Source | Pipeline | Management Responsibility | Frequency |
|---|---|---|---|
| Clearbit | |||
| CloudSQL Postgres | Stitch | Data Team | |
| DiscoverOrg | |||
| Gitter | |||
| GitLab dot Com | |||
| SheetLoad | SheetLoad | Data Team | |
| Marketo | Stitch | Data Team | 12 hour intervals - Backfilled from January 1, 2013 |
| Netsuite | Stitch | Data Team | 30 minute intervals - Backfilled from January 1, 2013 |
| Pings | Stitch/Custom | Data Team | |
| SFDC | Stitch | Data Team | 1 hour intervals - Backfilled from January 1, 2013 |
| Snowplow | Data Team | ||
| Zendesk | Stitch | Data Team | 1 hour intervals - Backfilled from January 1, 2013 |
| Zuora | Stitch | Data Team | 30 minute intervals - Backfilled from January 1, 2013 |
Planned:
This is the lag between real-time and the analysis displayed in the data visualization tool.
Process for adding a new data source:
gitlab-com/gl-securitySheetLoad is the process by which a GoogleSheet, local CSV or file from GCS can be ingested into the data warehouse.
As it is being iterated on often, the best place for up-to-date info on Sheetload is the Sheetload readme.
SheetLoad should only be used to bring in data whose canonical source is a spreadsheet, such as Sales quotas. You should never move data into a sheet or a csv for the expressed purpose of using SheetLoad to bring it into the warehouse. The minute who bring that data into the warehouse, it is already stale.
Except for where absolutely not possible, it is best that the SheetLoad sheet import from the original google sheet directly. Then you can format the sheetload version to be plain text. Any additional conversion can happen in the base dbt models.
In June of 2019, we switched sending Snowplow events from a third party to sending them to infrastructure managed by GitLab. From the perspective of the data team, not much changed from the third party implementation. Events are sent through the collector and enricher and dumped to S3. See Snowplow's architecture overview for more detail.
Enriched events are stored in TSV format in the bucket s3://gitlab-com-snowplow-events/output/. Bad events are stored as JSON in s3://gitlab-com-snowplow-events/enriched-bad/. For both buckets, there are paths that follow a date format of /YYYY/MM/DD/HH/<data>.
For details on how the ingestion infrastructure was set up, please see our Snowplow section of the handbook.
We are in the process of moving from GitLab CI to Airflow.
We currently use Snowflake as our data warehouse.
To gain access to the data warehouse:
Goal: Mitigate risk of people having access to sensitive data.
We currently use Meltano's Permission Bot in dry mode to help manage our user, roles, and permissions for Snowflake. Documentation on the permission bot is in the Meltano docs. Our configuration file for our Snowflake instance is stored in this config.yml file.
There are four things that we need to manage:
Two notes of the permission bot:
We currently use two databases- raw and analytics. The former is for EL'ed data; the latter is for data that is ready for analysis (or getting there).
analytics, analytics_staging, and analytics_sensitive. analytics_staging schema.emilie_scratch_analytics and emilie_scratch_staging) are also in the analytics database. The dev schemas could have sensitive information, as they iterate through it. These are not accessible within the GitLab Periscope workspace.Managing Roles for Snowflake
Here are the proper steps for provisioning a new user and user role:
securityadmin roleJSMITH - This is the GitLab default of first letter of first name and full last name.JSMITH) with sysadmin as the parent role (this grants the role to sysadmin)analytics and analytics_staging schemas to user with grant select on future tables in schema <schema> to role <username> using the sysadmin roleCompute resources in Snowflake are known as "warehouses". To better track and monitor our credit consumption, we have created several warehouses depending on who is accessing the warehouse. The names of the warehouse are appended with their size (analyst_s for small)
analyst_* - These are for Data Analysts to use when querying the database or modeling dataengineer_* - These are for Data Engineers and the Manager to use when querying the database or modeling datamerge_request_* - These are scoped to GitLab CI for dbt jobs within a merge requesttransforming_* - These are for production dbt jobsreporting - This is for the BI tool for queryingloading - This is for our Extract and Load jobsPlease see the data analyst onboarding issue template for details on getting started with dbt.
At times, we rely on dbt packages for some data transformation. Package management is built-in to dbt. A full list of packages available are on the dbt Hub site. We use the repository-syntax instead of the hub-syntax in our packages.yml file.
_xf dbt model should be a BEAM* table, which means it follows the business event analysis & model structure and answers the who, what, where, when, how many, why, and how question combinations that measure the business.accounts instead of accts.source table - (can also be called raw table) table coming directly from data source as configured by the manifest. It is stored directly in a schema that indicates its original data source, e.g. sfdcbase models- the only dbt models that reference the source table; base models have minimal transformational logic (usually limited to filtering out rows with data integrity issues or actively flagged not for analysis and renaming columns for easier analysis); can be found in the analytics_staging schema; is used in ref statements by end-user modelsend-user models - dbt models used for analysis. The final version of a model will likely be indicated with an _xf suffix when it’s goal is to be a BEAM* table. It should follow the business event analysis & model structure and answer the who, what, where, when, how many, why, and how question combinations that measure the business. End user models are found in the analytics schema.Schema References (aka What goes where)
| Purpose | Production | Dev | Config |
|---|---|---|---|
| For querying & analysis | analytics | emilie_scratch_analytics | {{ config({ "schema": "analytics", "post-hook": "grant select on {{this}} to role reporter" }) }} |
| staging | analytics_staging | emilie_scratch_staging | none |
| For querying & analysis but SENSITIVE | analytics_sensitive | emilie_scratch_analytics_sensitive | {{ config({ "schema": "sensitive_analysis", "post-hook": "grant select on {{this}} to role reporter_sensitive" }) }} |
| staging but SENSITIVE | analytics_staging_sensitive | emilie_scratch_analytics_staging_sensitive | {{ config({ "schema": "sensitive" }) }} |
~/.dbt/ folder there should be a profiles.ymlfile that looks like this sample profileSNOWFLAKE_TRANSFORM_WAREHOUSE as the variable name to identify the warehouse. The environment variable can be set in the .bashrc or .zshrc file as follows: export SNOWFLAKE_TRANSFORM_WAREHOUSE="ANALYST_XS"--vars '{warehouse_name: analyst_xl}' to the dbt commandWe use Periscope as our Data Visualization and Business Intelligence tool. To request access, please follow submit an access request.
The data team has a triager who is responsible for addressing dbt test failures; labeling, prioritizing, and asking initial questions on created issues; and guiding and responding to questions that come into the #data channel on Slack.
Having a dedicated triager on the team helps address the bystander affect. This is not an on-call position. This role just names a clear owner. Through clear ownership, we create room for everyone else on the team to spend most of the day around deep work. The triager is encouraged to plan their day for the kind of work that can be accomplished successfully with this additional demand on time.
The triager is not expected to know the answer to all the questions. They should pull in other team members who have more knowledge in the subject matter area to pass on the conversation after understanding the issue. Document any issues you stumble upon and learn about to help disseminate knowledge amongst all team members.
A team member who is off, on vacation, or working on a high priority project is responsible for finding coverage and communicating to the team who is taking over their coverage; this should be updated on the Data Team's Google Calendar.
dbt test failures Occasionally dbt test may produce failures. Many of these failures are upstream data quality problems that need to be addressed. Most test failures are documented in the dbt tests README. If you stumble upon one that isn't documented, be sure to document it to make it easier to address in the future.
Issue triaging Many issues that come into the data team project from other GitLab team members need additional info and/or context in order to be understood, estimated, and prioritized. It is the triagers priority to ask those questions. This will help folks feel like their asks are not just going to the void and are being taken seriously by the data team. It also help surface issues sooner, rather than later.
Slack requests Most of the requests that come in through Slack are solved by:
For all the same reasons as issue triaging, it's important that folks feel acknowledged.
Triage schedule The data team has implemented a triagle schedule that takes advantage of folks native timezones. This is also captured on the Data Team's Google Calendar.
| Local day | Team member |
|---|---|
| Monday (Sunday EST) | @amrvaljevich |
| Monday | @cathyr |
| Tuesday | @emilie |
| Wednesday | @tayloramurphy |
| Thursday | @tlapiana |
| Friday | @eli_kastelein |
