The mission of the Data Engineering team is to build a secure and trusted data platform that makes it possible for everyone to be an analyst so that our only limitations are the data or analysts themselves. We do this by means of our GitLab values and our Data Team Principles
Of the Data Team's Responsibilities the Data Engineering team is directly responsible for:
At GitLab we are Handbook First. Any changes to our codebase or process should have a handbook MR created before the work is started or concurrently with the MR to change the code.
For example: If we are adding a data source, we would first create the merge request to update our the extract and load and system diagram sections of our handbook. This MR can be updated as the change is created and merged along with the change.
In order to efficiently operate as a team we designate and assign the following responsibilities to members of the Data Engineering team. These assignments are not permanent, but do create a Directly Responsible Individual on the team for these responsibilities. The assigned Engineer is responsible for both maintenance and iterative improvement over these responsibilities.
Production operations is always our top standing priority on the Data Engineering team and includes all technical operations that support or affect our production environment and the SLOs it supports. This includes, but is not limited to monitoring, testing, deployment, code reviews, and DevOps generally.
The Engineer assigned the Project Management and Planning responsibility oversees, directs, and improves our workflow processes, milestone planning, and issue triaging. They ensure that work is getting planned and completed efficiently and that the work being done appropriately supports and enables the goals of both the Data Team and GitLab.
This engineer is responsible for the usability of our data platform. As stated in the Data Engineering mission, we are building the platform for everyone. The assigned Engineer proactively finds and solutions around our data platform with special mind to those experiences that are the most difficult to use.
This role keeps our platform secure, private, and compliant and will be the point of contact for audit questions as well as the primary engineer responsible for reviewing changes that affect the security and privacy of our data platform. They should also be iteratively improving the security features of our platform.
In order to keep our iterations inexpensive and increase the usability of our warehouse we have decided to implement a dimensional warehouse. This engineer is responsible for protecting the integrity of our warehouse architecture and leads the vision on where the architecture needs to be improved. All of these changes and protections should keep our customers, both internal an external, as their primary focus.